Antfly: Distributed, Multimodal Search and Memory and Graphs in Go!

Antfly represents a contemporary approach to information retrieval and knowledge management, integrating diverse search paradigms within a distributed system architecture. The system is engineered to address the growing complexity of data, particularly in contexts involving artificial intelligence and machine learning, by providing multimodal indexing, search, and memory capabilities. Its design emphasizes a unified operational model, from local development to scaled-out deployments, centered around a single-binary distribution.

Multimodal Search and Indexing Architecture

Traditional search engines often specialize in either full-text retrieval or structured data queries. The advent of machine learning, particularly deep learning models for embeddings, has introduced vector search as a third primary modality. Antfly integrates these distinct search methodologies—full-text, vector, and graph—into a cohesive system, allowing for complex queries that leverage the strengths of each. This multimodal approach is critical for applications requiring nuanced understanding of data, such as advanced Retrieval-Augmented Generation (RAG) systems or intelligent agents.

Full-Text Search

The full-text search component within Antfly functions akin to established inverted index systems. Documents are tokenized, normalized, and indexed to facilitate efficient keyword-based retrieval. This involves:

• Document Parsing: Extracting textual content from various input formats, including those derived from multimedia content through transcription or OCR.
• Tokenization and Analysis: Breaking down text into searchable units (tokens), followed by linguistic processing such as stemming, lemmatization, and stop-word removal.
• Inverted Index Construction: Mapping terms to the documents and their positions where they appear, enabling rapid lookups for textual relevance.

Antfly's full-text capabilities support standard query operators, Boolean logic, and relevance scoring, providing a robust foundation for traditional keyword-based information retrieval.

Vector Search

Vector search is foundational to Antfly's multimodal capabilities, enabling semantic similarity search across diverse data types. The process involves:

• Embedding Generation: Transforming raw data (text, images, audio, video) into high-dimensional numerical vectors, known as embeddings. This is often performed by pre-trained deep learning models. Antfly supports this natively via its Termite service or through external APIs.
• Vector Indexing: Storing these embeddings in specialized data structures optimized for Approximate Nearest Neighbor (ANN) search. Common ANN algorithms include Hierarchical Navigable Small World (HNSW), Locality Sensitive Hashing (LSH), or Product Quantization (PQ), which balance search speed with recall accuracy.
• Similarity Metrics: Employing distance functions (e.g., cosine similarity, Euclidean distance) to quantify the semantic resemblance between a query vector and indexed document vectors.

The integration of vector search allows Antfly to retrieve conceptually similar documents even when explicit keywords are absent, bridging the gap between lexical and semantic understanding.

Graph Search and Memory

Beyond textual and semantic similarity, Antfly incorporates graph capabilities to model and query relationships between documents or entities. This transforms the system into a form of "memory" or knowledge graph, where explicit connections can be established and traversed.

• Entity and Relationship Extraction: Identifying named entities (people, organizations, locations) and their relationships within documents, either through automated extraction or explicit definition.
• Graph Representation: Storing entities as nodes and relationships as edges in a graph structure. This allows for representing complex dependencies, hierarchies, and associative links.
• Traversal and Pattern Matching: Enabling queries that traverse these relationships, discover paths, or identify patterns within the graph. This facilitates contextual retrieval and inference, critical for sophisticated RAG applications that require understanding the "why" behind information, not just the "what."

A document in Antfly might be represented as a composite structure, encompassing its full text, its associated embedding vector, and its connections within the graph. A conceptual document structure could look like:

{
  "id": "doc-123",
  "content": "The quick brown fox jumps over the lazy dog.",
  "metadata": {
    "author": "john.doe",
    "timestamp": "2024-08-01T10:00:00Z",
    "source": "blog"
  },
  "text_embedding": [0.1, 0.2, ..., 0.9], // Vector for text content
  "image_embedding": [0.3, 0.4, ..., 0.7], // Vector for associated image
  "relationships": [
    { "type": "mentions", "target_id": "entity-fox" },
    { "type": "related_to", "target_id": "doc-456" }
  ]
}

Queries can then combine these modalities, for instance, "find documents semantically similar to this image AND containing the keyword 'fox' AND related to 'doc-456' in the knowledge graph." This synergy provides a powerful mechanism for complex information retrieval.

Distributed Systems Architecture

Antfly is engineered for high availability, scalability, and resilience through a distributed architecture built primarily in Go. Go's concurrency primitives (goroutines and channels), strong typing, and performance characteristics make it suitable for constructing robust distributed systems.

Multi-Raft Consensus

At the core of Antfly's distributed design is a multi-Raft consensus mechanism. Instead of a single, monolithic Raft group managing all state, Antfly leverages multiple independent Raft groups. This design choice offers significant advantages:

• Fault Isolation: Failure or performance degradation in one Raft group (e.g., due to a slow disk or network partition affecting a specific shard) does not necessarily impact other groups, improving overall system resilience.
• Parallelism: Operations pertaining to different data shards can proceed in parallel, as their consensus decisions are made independently by their respective Raft groups. This enhances throughput.
• Scalability: As the dataset grows, new Raft groups can be provisioned for new data shards, distributing the consensus workload across more nodes.

Antfly utilizes etcd's Raft library, a battle-tested and production-grade implementation of the Raft consensus algorithm. Each Antfly node participates in multiple Raft groups, acting as a follower, candidate, or leader for different groups concurrently.

The architecture separates metadata and data into distinct Raft groups:

• Metadata Raft Group(s): Responsible for maintaining cluster-wide configuration, schema definitions, shard mappings, and other control plane information. This ensures consistent views of the cluster state across all nodes.
• Data Raft Group(s): Each data shard (a partition of the overall document collection) is managed by its own Raft group. This group replicates write operations for that specific shard, ensuring data durability and consistency.

When a client initiates a write operation, a coordinator node determines the target data shard based on the document's properties. This request is then forwarded to the leader of the corresponding data Raft group, which replicates the change to its followers before committing.

Pebble Storage Engine

For persistent storage, Antfly integrates Pebble, CockroachDB's high-performance, LSM-tree (Log-Structured Merge-tree) inspired key-value store. Pebble is written in Go and designed for robust performance and durability.

• LSM-tree Advantages: LSM-trees are optimized for write-heavy workloads, common in database systems. They achieve high write throughput by buffering writes in memory (memtables) and periodically flushing them to immutable sorted string tables (SSTables) on disk. Reads traverse these layers, merging results.
• Durability and Atomicity: Pebble provides strong durability guarantees, often coupled with a write-ahead log (WAL) to ensure data recovery in case of system crashes. Its design supports atomic operations within a single key-value store instance.
• Integration with Raft: The Raft state machine on each node applies committed log entries to its local Pebble instance. This ensures that all replicas of a Raft group eventually converge to the same state. Each entry in the Raft log typically represents an operation (e.g., document insertion, update, deletion) that is then executed against Pebble.

The interaction can be conceptualized as:

graph TD
    Client --> CoordinatorNode
    CoordinatorNode --> RaftLeader[Raft Leader (Data Shard X)]
    RaftLeader -- Propose Log Entry --> RaftFollower1[Raft Follower (Data Shard X)]
    RaftLeader -- Propose Log Entry --> RaftFollower2[Raft Follower (Data Shard X)]
    RaftLeader -- Apply Committed Entry --> PebbleLeader[Pebble (Leader)]
    RaftFollower1 -- Apply Committed Entry --> PebbleFollower1[Pebble (Follower 1)]
    RaftFollower2 -- Apply Committed Entry --> PebbleFollower2[Pebble (Follower 2)]

Sharding Strategy

Antfly employs a sharding strategy to distribute data across the cluster. Documents are partitioned based on a sharding key (e.g., document ID hash, or a user-defined key), with each shard managed by a dedicated Raft group.

• Data Sharding: Distributes the document corpus across multiple nodes, reducing the amount of data any single node needs to manage and index. This improves query performance and write throughput.
• Metadata Sharding: While the core metadata might be replicated across a primary metadata Raft group, larger, less frequently accessed metadata (e.g., detailed schema for specific tenants) could also be sharded and managed by separate groups, though the prompt primarily indicates "metadata and data shards get their own Raft groups" implying a separation at that level.

Single Binary Deployment

The antfly swarm command provides a single-process deployment, encompassing all necessary components (Raft nodes, storage engine, search indexes, Termite ML service). This simplifies local development, testing, and small-scale deployments significantly, as it eliminates complex setup procedures. For production, additional nodes can be seamlessly added to an existing swarm deployment, allowing for horizontal scaling and increased resilience.

Native ML Inference with Termite

A distinguishing feature of Antfly is its integrated ML inference service, named Termite. This service allows Antfly to perform crucial machine learning tasks locally, reducing reliance on external AI inference APIs and services.

Purpose and Functionality

Termite is designed to act as an internal Ollama-like service, but specifically for non-generative models and some generative tasks relevant to data processing. Its primary functions include:

• Embeddings Generation: Converting text, images, audio, and video into vector embeddings for vector search. This is crucial for multimodal capabilities.
• Reranking: Improving the relevance of search results by re-scoring them using more sophisticated ML models.
• Chunking: Breaking down large documents into smaller, manageable chunks suitable for processing by LLMs or for more granular retrieval.
• Text Generation (Limited): While not its primary focus, Termite can perform text generation tasks, likely for summarization, entity extraction, or other non-chat-oriented generative applications.

Operational Advantages

The native integration of Termite offers several key benefits:

• Reduced Latency: Inference occurs within the same operational boundary as the data, minimizing network overhead and API call latency.
• Cost Efficiency: Eliminates recurring costs associated with external API usage, making ML-powered features more economical.
• Offline Capability: Allows Antfly to perform ML inference even in environments with limited or no external network access, crucial for edge deployments or sensitive data handling.
• Data Locality and Security: Data does not leave the Antfly cluster for embedding generation or other inference tasks, enhancing data privacy and compliance.
• Simplified Deployment: No separate inference service needs to be managed or scaled independently, streamlining operations.

Model Management

Termite manages a registry of models, which can be loaded and unloaded as needed. While specific details of model loading (e.g., ONNX, GGML, custom formats) are not provided, the intent is clear: to offer a flexible, in-process inference runtime. Developers can configure which models Termite should use for different tasks.

// Conceptual Go interface for Termite's embedding service
type EmbeddingService interface {
    GenerateTextEmbedding(text string, modelID string) ([]float32, error)
    GenerateImageEmbedding(imageData []byte, modelID string) ([]float32, error)
    // ... other multimodal embedding methods
}

// Example usage within Antfly's indexing pipeline
func (i *Indexer) indexDocument(doc Document) error {
    // ... process text content ...
    textEmbedding, err := i.termite.GenerateTextEmbedding(doc.Content, "bge-small-en-v1.5")
    if err != nil {
        return fmt.Errorf("failed to generate text embedding: %w", err)
    }
    doc.TextEmbedding = textEmbedding

    // ... process image content if present ...
    if doc.HasImage() {
        imageEmbedding, err := i.termite.GenerateImageEmbedding(doc.ImageData, "clip-vit-base-patch32")
        if err != nil {
            return fmt.Errorf("failed to generate image embedding: %w", err)
        }
        doc.ImageEmbedding = imageEmbedding
    }

    // ... store document with embeddings in Pebble ...
    return i.storage.Store(doc)
}

Despite its native capabilities, Antfly maintains support for external ML inference providers (OpenAI, Ollama, Bedrock, Gemini, etc.). This provides flexibility, allowing users to leverage specialized models or existing infrastructure if desired, or to offload heavy inference tasks to dedicated services.

Data Model and Operations

Antfly's data model and operational capabilities are designed to support dynamic and real-time use cases, particularly in the context of streaming data and iterative RAG processes.

MongoDB-style In-place Updates

Antfly supports "MongoDB-style" in-place updates, which typically refers to the ability to modify specific fields within a document without needing to replace the entire document. This offers several benefits:

• Reduced Bandwidth: Only the changed parts of a document need to be transmitted and processed.
• Atomic Updates: Can enable atomic modifications to sub-fields, improving consistency for concurrent operations.
• Efficiency for Partial Changes: Common in applications where documents are frequently updated incrementally (e.g., user profiles, evolving entity states).

While "in-place" for a traditional B-tree might mean direct modification, in an LSM-tree like Pebble, an update conceptually translates to writing a new version of the document (or the modified parts) to the highest-level memtable. The LSM-tree's merge-on-read mechanism ensures that the latest version of a document is always retrieved. The "in-place" abstraction simplifies the application logic, allowing developers to think about document mutations rather than versioning or full document replacement. Antfly's implementation likely manages the underlying key-value operations to make these partial updates efficient.

Streaming RAG

Streaming RAG (Retrieval-Augmented Generation) refers to scenarios where information retrieval is tightly integrated into a real-time LLM interaction, potentially providing continuously updated context or processing queries in a streaming fashion. Antfly facilitates this through:

• Low-Latency Retrieval: Its distributed architecture, optimized indexes, and native ML inference contribute to rapid information retrieval, essential for real-time applications.
• Efficient Indexing of New Data: Antfly's ability to ingest and index new documents efficiently, coupled with its update mechanisms, ensures that the knowledge base used for RAG is current.
• Modular Query Composition: The multimodal search capabilities allow for dynamic query construction based on ongoing conversational context, enabling LLMs to intelligently retrieve relevant information as needed.

Consider a scenario where an LLM is engaging in a dialogue. As the conversation progresses, new entities or topics emerge. Antfly can:

1. Receive a query (e.g., current conversation turn).
2. Generate a vector embedding for the query using Termite.
3. Perform a vector search to find semantically similar documents.
4. Optionally, perform a full-text search for specific keywords or a graph traversal for related concepts.
5. Return relevant chunks of information to the LLM as context, potentially in a streaming fashion, allowing the LLM to start generating a response even before all context is fully retrieved.

Ecosystem and Deployment

Antfly's ecosystem components aim to provide a comprehensive solution for operating and integrating it within modern cloud-native environments and AI workflows.

Kubernetes Operator

A Kubernetes Operator streamlines the deployment, management, and scaling of Antfly clusters within a Kubernetes environment. The operator automates tasks such as:

• Provisioning: Deploying Antfly nodes, configuring their Raft groups, and setting up storage.
• Scaling: Adjusting the number of nodes or shards based on demand.
• Updates and Upgrades: Performing rolling updates to new Antfly versions with minimal downtime.
• Backup and Restore: Orchestrating data backup and recovery procedures.
• Monitoring: Exposing metrics for health checks and performance analysis.

This significantly reduces the operational burden of running a distributed database, aligning Antfly with cloud-native best practices.

MCP Server for LLM Tool Use

Antfly includes an MCP (Model-Controller-Program) server, designed to expose Antfly's capabilities as tools for Large Language Models. This positions Antfly as a critical component in the emerging paradigm of LLM-powered agents and workflows.

• Tool Abstraction: The MCP server abstracts Antfly's search, memory, and data manipulation functions into a set of callable tools that an LLM can understand and invoke.
• Structured Communication: It provides a structured interface (e.g., JSON or gRPC based) through which an LLM can formulate requests to Antfly (e.g., "search for documents about X," "retrieve relationships for entity Y," "update document Z") and receive structured results.
• Enabling Agentic Behavior: By giving LLMs access to Antfly as a tool, they can perform actions like searching for facts, recalling past interactions (memory), or updating their knowledge base, moving beyond simple conversational responses to more autonomous, goal-oriented behavior.

For example, an LLM agent could receive a complex question: "What are the latest research papers from [researcher X] related to [topic Y], and who are their key collaborators?" The LLM, via the MCP server, could:

1. Invoke Antfly's graph search to find researcher X's collaborators.
2. Invoke Antfly's multimodal search (full-text + vector) to find recent papers by researcher X and related to topic Y.
3. Synthesize the results to answer the original question.

Licensing Considerations

Antfly is licensed under the Elastic License v2 (ELv2). This license choice is significant and reflects a particular stance on open-source sustainability within the commercial landscape.

• Permissiveness: Users are permitted to use, modify, and self-host Antfly for their own internal operations. They can also build products and applications that utilize Antfly as an underlying component.
• Restriction: The primary restriction is that users are explicitly prohibited from offering Antfly itself as a managed service. This prevents other entities from directly competing with the project's developers by offering Antfly as a commercial SaaS product without contributing back financially or collaboratively.

This licensing model aims to protect the sustainability of the project while keeping the source code publicly available for inspection, modification, and self-hosting by the broader developer community. It represents a compromise between fully open-source (OSI-approved) licenses and proprietary commercial licenses.

Conclusion

Antfly presents a robust and forward-looking solution for distributed multimodal search and memory, particularly well-suited for the demands of modern AI-driven applications. Its architectural choices—combining a multi-Raft consensus layer over Pebble, multimodal indexing, and native ML inference via Termite—demonstrate a clear intent to provide a high-performance, scalable, and operationally efficient platform. The integrated ecosystem tools, such as the Kubernetes Operator and MCP server, further solidify its position as a comprehensive solution for managing complex, interconnected data and enabling advanced LLM capabilities. The design principles emphasize a unified, single-binary deployment for simplicity at smaller scales, with clear pathways for horizontal scaling and integration into sophisticated cloud-native and AI agent workflows.

For further exploration of distributed systems design, multimodal search strategies, or specialized consulting services related to high-performance data infrastructure, please visit https://www.mgatc.com.

---

Originally published in Spanish at www.mgatc.com/blog/antfly-distributed-multimodal-search-memory-graphs-go/