DEV Community: Actian for Developers

Build a Local RAG Pipeline on Declassified UFO Files with VectorAI DB and Ollama

Oluseye Jeremiah — Mon, 20 Jul 2026 14:45:31 +0000

Production document collections rarely look like benchmark datasets. Some PDFs contain searchable text. Others are scanned pages that need OCR. Metadata varies across sources, and retrieval pipelines that work on clean examples often struggle once those differences appear.

In this tutorial, you'll build a fully local RAG pipeline around declassified U.S. government UFO files from the PURSUE archive. Using Ollama and VectorAI DB, you'll extract text, handle scanned documents, attach metadata, and generate cited answers without relying on external APIs.

The UFO files are simply the dataset. The same pipeline applies to internal documentation, legal discovery archives, and research repositories where documents come from different sources and follow different formats.

By the end, you'll have a single-machine RAG system that ingests PDFs, stores embeddings with metadata, filters results by agency or decade, and generates responses grounded in the source documents.

What You Are Building

The system has three layers, all running on a single machine.

Ingestion: PDFs from the PURSUE archive pass through an ingestion pipeline that handles both machine-readable and scanned documents. PyMuPDF extracts text where possible, while pytesseract processes image-based pages. The text is chunked, embedded locally with Ollama's nomic-embed-text, and stored in VectorAI DB with metadata: agency, decade, source filename, and source URL.

Retrieval: Queries are embedded with the same model used during ingestion. VectorAI DB returns the nearest chunks and can narrow the search using metadata filters before ranking by similarity.

Generation: Retrieved chunks and the user's question are sent to llama3.2, which generates an answer grounded in the retrieved documents and includes references back to the original files.

Everything runs locally on a single machine. A GPU speeds up embedding and generation, but the pipeline works on CPU-only hardware.

Figure 1: System architecture diagram. Three layers stacked vertically inside a single machine boundary.

Set Up the Environment

Get VectorAI DB and Ollama running, then install the Python dependencies.

Create a docker-compose.yml file in your project folder:

services:
  vectorai:
    image: actian/vectorai:latest
    container_name: vectorai-ufo
    ports:
      - "6573:6573"
      - "6574:6574"
      - "6575:6575"
    volumes:
      - ./local_data:/var/lib/actian-vectorai
    environment:
      - ACTIAN_VECTORAI_ACCEPT_EULA=YES
    restart: unless-stopped

Start it:

docker-compose up -d

The image exposes REST on port 6573, gRPC on port 6574, and a local web UI on 6575. Verify the database is running:

docker logs vectorai-ufo

You should see Ready to accept connections... near the end of the log, along with your license tier and current vector count:

Figure 2: Docker logs

Install Ollama from ollama.com/download. Pull both models:

ollama pull nomic-embed-text
ollama pull llama3.2

Install Python dependencies:

pip install actian-vectorai-client pymupdf pytesseract ollama

Download and Inspect the Document Collection

The PURSUE Release 01 is available directly at war.gov/UFO/. The release contains 162 files: 120 PDFs, 28 videos, and 14 high-resolution images. This tutorial ingests the PDFs only. A second release on May 23, 2026 added 64 more files.

For faster onboarding, the community project DenisSergeevitch/UFO-USA has all 120 PDFs converted to Markdown via Gemini OCR, totaling 4,185 pages. Clone it to skip the OCR step during development:

git clone https://github.com/DenisSergeevitch/UFO-USA.git mirror

This tutorial reads from the Markdown archive but also shows the PyMuPDF and pytesseract code so you understand how to handle raw PDFs on your own document collection.

The PURSUE collection mixes two PDF types. State Department memos and NARA records have extractable text layers and parse cleanly with PyMuPDF. The major FBI case file (62-HQ-83894, covering UFO investigations from 1947 to 1968, totaling over 1,100 pages across multiple sections) consists of scanned images and requires OCR. Your ingestion pipeline branches on PDF type because the same code has to handle both.

Figure 3: Side-by-side comparison of a machine-readable State Department PDF page and a scanned FBI case file page, showing why the OCR branch exists.

Build the Ingestion Pipeline

Walk through ingestion in six steps. Every code block runs.

Step 1 Detect the PDF type: Try to extract text with PyMuPDF first. If the page yields fewer than 50 characters, treat it as scanned and route to OCR.

import fitz  # PyMuPDF
import pytesseract
from PIL import Image
import io
def extract_pdf_text(path: str) -> str:
    """Extract text from a PDF, routing scanned pages through OCR."""
    doc = fitz.open(path)
    all_text = []

    for page in doc:
        text = page.get_text().strip()

        # Fewer than 50 chars on a non-empty page usually means scanned.
        if len(text) < 50:
            pix = page.get_pixmap(dpi=200)
            img = Image.open(io.BytesIO(pix.tobytes("png")))
            text = pytesseract.image_to_string(img)

        all_text.append(text)

    doc.close()
    return "\n\n".join(all_text)

Step 2. Chunk the text: Use 700-word chunks with 70-word overlap. This stays comfortably within the embedding model's context window while preserving enough surrounding text to keep related ideas together during retrieval.

def chunk_text(text: str, size: int = 700, overlap: int = 70):
    """Split text into overlapping word chunks."""
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = " ".join(words[i:i + size])
        if chunk.strip():
            chunks.append(chunk)
        i += size - overlap
    return chunks

Step 3. Embed each chunk: Call Ollama's embeddings endpoint with nomic-embed-text. The model returns a 768-dimensional vector for each chunk.

import ollama
def embed(text: str):
    """Return a 768-dim embedding for the input text."""
    resp = ollama.embeddings(model="nomic-embed-text", prompt=text)
    return resp["embedding"]

Step 4. Create the VectorAI DB collection: The schema holds the 768-dimensional vector plus metadata fields for filtering: agency, decade, document type, source filename, and the direct war.gov URL for the source PDF.

from actian_vectorai import VectorAIClient, VectorParams, Distance

COLLECTION = "pursue_demo"

with VectorAIClient("localhost:6574") as client:
    client.collections.get_or_create(
        name=COLLECTION,
        vectors_config=VectorParams(size=768, distance=Distance.Cosine),
    )

API method names and configuration keys in this tutorial are verified against docs.vectoraidb.actian Check the reference before deploying to production, as signatures may change between SDK versions.

Step 5. Classify each document by agency and decade: Derive these from the filename and store them in the payload so retrieval can filter on them later.

import re
def classify_agency(filename: str) -> str:
    name = filename.lower()
    if "fbi" in name or "62-hq" in name:
        return "FBI"
    if "nasa" in name:
        return "NASA"
    if "state" in name or "342" in name:
        return "State"
    if "odni" in name:
        return "ODNI"
    return "Unknown"

def derive_decade(filename: str) -> int:
    years = re.findall(r"(19[4-9][0-9]|20[0-2][0-9])", filename)
    if not years:
        return 0
     return (int(years[0]) // 10) * 10

Step 6. Insert chunks with metadata: Keep the client connection open for the full ingestion run. Opening and closing a connection per document adds overhead and can trigger server-side keepalive limits on long embedding passes.

import os
import uuid
from pathlib import Path
from actian_vectorai import PointStruct

def ingest_document(client, text: str, doc_name: str, source_url: str = ""):
    chunks = chunk_text(text)
    agency = classify_agency(doc_name)
    decade = derive_decade(doc_name)

    points = []
    for chunk in chunks:
        vector = embed(chunk)
        points.append(PointStruct(
            id=uuid.uuid4().int >> 64,
            vector=vector,
            payload={
                "content": chunk,
                "agency": agency,
                "decade": decade,
                "document_type": "scanned" if agency == "FBI" else "machine_readable",
                "source_doc": doc_name,
                "source_url": source_url,
            },
        ))

    if points:
        client.points.upsert(COLLECTION, points=points)
    return len(points)


# Run ingestion with a single open connection
with VectorAIClient("localhost:6574") as client:
    total = 0
    converted = Path("mirror/converted")
    for doc_name in sorted(os.listdir(converted)):
        doc_dir = converted / doc_name
        if not doc_dir.is_dir():
            continue
        full_text = ""
        for page_file in sorted(doc_dir.glob("page-*.md")):
            full_text += page_file.read_text(encoding="utf-8") + "\n\n"
        total += ingest_document(client, full_text, doc_name)
    client.vde.flush(COLLECTION)
    print(f"Done. {total} chunks inserted.")

Figure 4: ingestion process

client.vde is the Vector Data Engine handle for low-level storage operations. Calling flush() after the ingestion loop ensures all writes are committed before the session ends, preventing data loss if the process stops immediately after ingestion.

Run Your First Query

Retrieval works by embedding the query with the same model used at ingestion time, then finding the closest vectors in the collection.

def retrieve(query: str, k: int = 5):
    query_vector = embed(query)
    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
        ) or []


results = retrieve("flying disc sightings reported to government agencies")

for r in results:
    p = r.payload
    print(f"score={r.score:.3f}  agency={p['agency']:<8}  doc={p['source_doc'][:50]}")
    print(f"  {p['content'][:200]}...\n")

Running this returns:

Figure 5: Retrieval process
The query retrieves both State Department records and FBI files, even though the wording differs across documents. Retrieval is based on semantic similarity rather than exact keyword matches, allowing related material from different agencies to surface together.

Figure 6: Query top 5 results

Add Metadata Filters

Metadata filtering runs before similarity ranking, so only documents matching the filter are scored. The same query returns different document sets depending on which agency you scope it to.

from actian_vectorai import Field, FilterBuilder

def retrieve_filtered(query: str, agency: str = None, decade: int = None, k: int = 5):
    query_vector = embed(query)
    builder = FilterBuilder()
    if agency:
        builder = builder.must(Field("agency").eq(agency))
    if decade:
        builder = builder.must(Field("decade").eq(decade))
    filter_obj = builder.build() if (agency or decade) else None

    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
            filter=filter_obj,
        ) or []

Run the same query three ways:
# Unfiltered: results across all agencies
all_results = retrieve_filtered("flying disc sighting")

# FBI documents only
fbi_results = retrieve_filtered("flying disc sighting", agency="FBI")

# State Department documents only
state_results = retrieve_filtered("flying disc sighting", agency="State")

On a company knowledge base you'd filter by department or document type. On a legal discovery archive you'd filter by custodian or date range. The pattern is the same.

Figure 7: Query Results

Wire In the Local LLM

Connect retrieval to generation. The LLM receives the user's question and the retrieved chunks, and produces a grounded answer with citations.

def answer(query: str, k: int = 5, agency: str = None):
    results = retrieve_filtered(query, agency=agency, k=k)

    context_blocks = []
    for i, r in enumerate(results, 1):
        p = r.payload
        context_blocks.append(
            f"[Source {i}] Agency: {p['agency']} | Document: {p['source_doc']}\n"
            f"{p['content']}\n"
        )
    context = "\n---\n".join(context_blocks)

    prompt = f"""Answer the question using only the sources below. Cite each claim \
by source number in square brackets, like [Source 2]. If the sources do not answer \
the question, say so.

SOURCES:
{context}

QUESTION: {query}

ANSWER:"""

    response = ollama.generate(model="llama3.2", prompt=prompt, stream=True)
    for chunk in response:
        print(chunk["response"], end="", flush=True)
    print()

    print("\nSources cited:")
    for i, r in enumerate(results, 1):
        p = r.payload
        print(f"  [{i}] {p['agency']:<8} {p['source_doc']}")
        if p.get("source_url"):
            print(f"         {p['source_url']}")

Running the pipeline end-to-end against the 1949 State Department documents:

Query: What did witnesses report about flying discs in 1949?

Figure 8:LLM reuslts

Each statement in the answer comes from retrieved documents. The source links make it possible to verify the underlying material directly. The answer cites specifics: delta-shaped, no banking on the 180-degree turn, surface color between light gray and dirty white. These details come from the 1949 USAF intelligence memo, not from the model's training data. The whole response ran on a laptop with no API key required. Generation time depends on your CPU and the response length. Expect anywhere from 30 seconds to a few minutes on CPU-only hardware.

Three Substitution Paths

The architecture is not specific to the PURSUE files.

Swap the document collection: Replace the PURSUE files with company documentation, legal records, clinical trial archives, or any other PDF collection. The ingestion pipeline, VectorAI DB schema, and retrieval logic stay the same. Update classify_agency and derive_decade to extract whatever metadata matters for your collection: department, project ID, jurisdiction, author. Anything stored in the payload at ingestion time becomes a filter at retrieval time.

Scale to the full document collection: This tutorial demonstrates the architecture on a focused subset. The GitHub repository includes a full ingestion script that processes all 120 PDFs across the five agencies in PURSUE Release 01. On CPU, the full run takes several hours. On a GPU-enabled machine, the embedding pass takes under 30 minutes. The Community Edition license supports up to 5,000 vectors, which covers the full Release 01 collection at these chunk settings. Release 02 (May 23, 2026) added 64 more files. A scheduled job that runs the ingestion pipeline on new downloads keeps the collection current.

Deploy on any infrastructure: The Docker Compose file runs identically on a laptop, a server, an edge device, or an air-gapped environment. VectorAI DB is the same across all of them. The hardware changes, not the architecture or the application code.

Wrapping Up

You built a fully local RAG pipeline that handles mixed-format documents, stores embeddings with metadata, retrieves relevant chunks, and generates cited answers using Ollama and VectorAI DB.

The UFO archive is only one example. The same approach applies to company documentation, legal records, research repositories, and other collections where documents arrive from different sources and follow different formats.

In practice, messy document collections are often the hard part of RAG systems. Building a pipeline that can handle those differences matters more than the dataset itself.

The complete code, Docker Compose configuration, and requirements are available in the GitHub repository. The accompanying README includes instructions for processing the full PURSUE collection and adapting the pipeline to other document libraries.

For deeper VectorAI DB implementation details, see the VectorAI DB Python SDK reference.

Build a Local Document Intelligence Agent with LlamaIndex and VectorAI DB

Odewole Babatunde Samson — Thu, 16 Jul 2026 13:47:11 +0000

Most LlamaIndex vector store integrations assume an outbound connection. Pinecone, Weaviate, Qdrant Cloud, and the rest of the officially supported list all require a network call to an external service. That works fine until it doesn't, until the deployment target is air-gapped, the data classification prohibits external API calls, or the project simply cannot take on a cloud dependency.

This tutorial integrates Actian VectorAI DB with LlamaIndex via the custom VectorStore interface and builds a document intelligence agent on that foundation. Every component runs on local hardware: LlamaIndex for orchestration, nomic-embed-text via Ollama for embeddings, VectorAI DB for retrieval, and a local Large Language Model (LLM ) via Ollama for generation. Nothing leaves the machine.

The integration uses ActianVectorAIVectorStore from the official llama-index-vector-stores-actian-vectorai package, the same interface pattern every other LlamaIndex integration follows, and ships in the GitHub repo as a standalone module importable into any existing LlamaIndex project.

How LlamaIndex's VectorStore Interface Works

Before writing implementation code, orient yourself to the interface. If you have already used LlamaIndex's built-in integrations, you will recognize this pattern immediately because every integration in the ecosystem implements the same four methods.

LlamaIndex's VectorStore base class requires:

add: Accept a list of BaseNode objects. Extract the embedding and metadata from each node. Insert them into the target database. Return the list of inserted node IDs.
delete: Accept a node ID. Remove the corresponding record from the target database.
query: Accept a VectorStoreQuery object containing the query embedding, top-k value, and optional metadata filters. Execute a vector search. Return a VectorStoreQueryResult.
get_nodes: Accept a list of node IDs. Retrieve the corresponding records from the target database. Return them as BaseNode objects.

Every official LlamaIndex vector store integration implements these four methods against its target database's SDK.

Here are the method signatures from the LlamaIndex VectorStore base class:

# LlamaIndex VectorStore base class method signatures

from llama_index.core.vector_stores.types import (
    VectorStore,
    VectorStoreQuery,
    VectorStoreQueryResult,
)
from llama_index.core.schema import BaseNode
from typing import Any, List, Optional


class VectorStore:
    """Base class. Your implementation must override these four methods."""

    def add(self, nodes: List[BaseNode], **add_kwargs: Any) -> List[str]:
        """
        Insert nodes with their embeddings into the vector store.
        Returns a list of inserted node IDs.
        """
        raise NotImplementedError

    def delete(self, ref_doc_id: str, **delete_kwargs: Any) -> None:
        """
        Delete nodes associated with the given document ID.
        """
        raise NotImplementedError

    def query(
        self,
        query: VectorStoreQuery,
        **kwargs: Any,
    ) -> VectorStoreQueryResult:
        """
        Execute a vector similarity search.
        Returns top-k nodes matching the query embedding.
        """
        raise NotImplementedError

    def get_nodes(
        self,
        node_ids: Optional[List[str]] = None,
        filters: Optional[Any] = None,
    ) -> List[BaseNode]:
        """
        Retrieve nodes by their IDs.
        Returns a list of BaseNode objects.
        """
        raise NotImplementedError

Your VectorAI DB integration implements these four methods using VectorAIClient from the Python SDK. That is the complete scope of the implementation.

What You Are Building

Here is the full agent architecture before any implementation code.

Your agent ingests a local PDF document library using LlamaIndex's SimpleDirectoryReader. Documents are split into nodes using LlamaIndex's SentenceSplitter with a 512-token chunk size and 50-token overlap.

Each node is embedded using nomic-embed-text, which runs via Ollama and produces 768-dimensional vectors. Those embeddings are stored in the VectorAI DB through the custom VectorStore integration you will build in this tutorial.

At query time, the agent embeds your question with the same nomic-embed-text model, retrieves the top-5 most similar nodes from VectorAI DB, and passes them to a local LLM via Ollama as context for generation. The LLM cites the source chunks in its response.

Figure 1: The full agent architecture

Hardware baseline

16 GB RAM, 4-core CPU. The full stack runs without a GPU on standard developer hardware. Generation is slower without a GPU, but fully functional.

Prerequisites

To follow along, install the following tools:

Docker and Docker Compose
Python 3.10 or higher
PIP or UV

Set Up the Environment

Get all dependencies running before writing any integration code. Every command below runs without modification on a standard Linux or macOS machine.

Set up the project with UV.

uv init llamaindex-vectoraidb-agent
cd llamaindex-vectoraidb-agent
mkdir -p docs

Start VectorAI DB with Docker Compose

Create docker-compose.yml:

services:
  vectoraidb:
    image: actian/vectorai:latest
    container_name: vectorai
    ports:
      - "6573:6573"
      - "6574:6574"
    volumes:
      - vectoraidb_storage:/var/lib/actian-vectorai
    environment:
      - ACTIAN_VECTORAI_ACCEPT_EULA=YES
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6574/health"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  vectoraidb_storage:

Start the service:

docker compose up -d

Expected startup output:

Figure 2: VectorAI DB Docker service running

Wait for the health check to pass, then confirm VectorAI DB is running:

curl localhost:6574/health

Expected response:

{"status":"ok"}

Install Ollama and pull the required models.

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull the embedding model
ollama pull nomic-embed-text

# Pull the generation model
ollama pull llama3.2

Add all dependencies:

uv add \
  llama-index-vector-stores-actian-vectorai \
  llama-index-core \
  llama-index-llms-ollama \
  llama-index-embeddings-ollama \
  pymupdf

uv add resolves, pins, and installs everything in one step. The llama-index-vector-stores-actian-vectorai package pulls in actian-vectorai-client automatically, that is the SDK that ships VectorAIClient and AsyncVectorAIClient.

Your pyproject.toml will contain:

[project]
name = "llamaindex-vectoraidb-agent"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "llama-index-core>=0.14.22",
    "llama-index-embeddings-ollama>=0.9.0",
    "llama-index-llms-ollama>=0.10.1",
    "llama-index-vector-stores-actian-vectorai>=0.1.0",
    "pymupdf>=1.27.2.3",
]

Your environment is ready. VectorAI DB is running on port 6574, Ollama is serving both models, and all Python packages are installed.

Build the VectorAI DB VectorStore Integration

The integration ships in the official llama-index-vector-stores-actian-vectorai package as ActianVectorAIVectorStore. There is no custom class to write. The module below re-exports it from a single local path, so every script in the project imports consistently:

# vectoraidb_vectorstore.py -- integration entry point
"""
vectoraidb_vectorstore.py
--------------------------
LlamaIndex VectorStore integration for Actian VectorAI DB.

Re-exports ActianVectorAIVectorStore from the official package so any
script in this project imports from a single location.

    from vectoraidb_vectorstore import ActianVectorAIVectorStore

API reference: https://docs.vectoraidb.actian.com/docs/integrations/llama-index
SDK clients:   actian_vectorai.VectorAIClient  (sync)
               actian_vectorai.AsyncVectorAIClient  (async)
Default port:  6574
"""

from llama_index.vector_stores.actian_vectorai import ActianVectorAIVectorStore

__all__ = ["ActianVectorAIVectorStore"]

Drop this file into any existing LlamaIndex project. The import path remains the same regardless of where ActianVectorAIVectorStore is located within the official package in future versions.

Key constructor parameters, verified against VectorAI DB before deploying to production:

Parameter	Default	Notes
`url`	`"localhost:6574"`	`host:port` format, no `http://` scheme
`collection_name`	`"llama_index_collection"`	Created automatically on first use
`dense_vector_name`	`"llama_index_dense_vector"`	Name of the dense vector field inside the collection
`dense_vector_params`	`None`	Inferred from first embedding when omitted; defaults to cosine distance
`stores_text`	`False`	Set `True` to store node text in the point payload for retrieval
`clear_existing_collection`	`False`	Delete the existing collection before the first operation
`client`	`None`	Pass a pre-configured `VectorAIClient` to manage connection lifecycle externally
`async_client`	`None`	Pass a pre-configured `AsyncVectorAIClient`

The class supports three connection patterns: context manager (recommended), manual connect()/close(), and an external client. All three are shown in the scripts below.

Ingest Your Documents

Use the context manager pattern to load a local PDF document library into VectorAI DB. The context manager calls connect() on entry and shutdown() on exit, so the connection lifecycle is handled automatically.

# ingest.py -- full ingestion pipeline

import argparse
import sys

from actian_vectorai import VectorAIClient
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    Settings,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama

from vectoraidb_vectorstore import ActianVectorAIVectorStore

VECTORAIDB_URL  = "localhost:6574"   # host:port -- no http:// scheme
COLLECTION_NAME = "document_intelligence"
DOCS_DIR        = "./docs"
EMBED_MODEL     = "nomic-embed-text"
LLM_MODEL       = "llama3.2"
CHUNK_SIZE      = 512
CHUNK_OVERLAP   = 50


def parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="Ingest PDFs into VectorAI DB")
    p.add_argument("--docs",       default=DOCS_DIR,        help="Directory of PDFs")
    p.add_argument("--collection", default=COLLECTION_NAME, help="VectorAI DB collection name")
    p.add_argument("--url",        default=VECTORAIDB_URL,  help="VectorAI DB host:port")
    return p.parse_args()


def main() -> None:
    args = parse_args()

    Settings.embed_model = OllamaEmbedding(
        model_name=EMBED_MODEL,
        base_url="http://localhost:11434",
    )
    Settings.llm = Ollama(
        model=LLM_MODEL,
        base_url="http://localhost:11434",
        temperature=0,
        request_timeout=120.0,
    )
    Settings.node_parser = SentenceSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
    )

    print(f"Loading documents from '{args.docs}' ...")
    try:
        documents = SimpleDirectoryReader(
            input_dir=args.docs,
            required_exts=[".pdf"],
            recursive=True,
        ).load_data()
    except Exception as e:
        print(f"\nERROR: Could not load documents from '{args.docs}'\n  {e}")
        print("Fix: put at least one PDF in the docs/ directory.")
        sys.exit(1)

    if not documents:
        print(f"No PDF files found in '{args.docs}'. Add some PDFs and re-run.")
        sys.exit(1)

    print(f"Loaded {len(documents)} document pages.")
    print(f"Connecting to VectorAI DB at {args.url} ...")

    try:
        # Context manager pattern: connect() on __enter__, shutdown() on __exit__.
        # dense_vector_params is omitted -- vector size is inferred from the first
        # inserted embedding and distance defaults to cosine.
        with ActianVectorAIVectorStore(
            url=args.url,
            collection_name=args.collection,
            stores_text=True,   # store node text in payload for retrieval
        ) as vector_store:

            storage_context = StorageContext.from_defaults(vector_store=vector_store)

            print("Building VectorStoreIndex (embedding + inserting into VectorAI DB) ...")
            VectorStoreIndex.from_documents(
                documents,
                storage_context=storage_context,
                show_progress=True,
            )

            # Verify: check point count directly via the underlying VectorAIClient
            client: VectorAIClient = vector_store.client
            info = client.get_collection(args.collection)
            print(f"\nIngestion complete.")
            print(f"Collection '{args.collection}': {info.points_count} points in VectorAI DB.")

    except ConnectionRefusedError:
        print(f"\nERROR: Cannot reach VectorAI DB at {args.url}")
        print("Fix: run 'docker compose up -d' and wait for the health check to pass.")
        sys.exit(1)


if __name__ == "__main__":
    main()

Run it:

uv run ingest.py
# or point at a different directory:
uv run ingest.py --docs /path/to/your/pdfs --collection my_collection

Expected output:

Loading documents from './docs' ...
Loaded 87 document pages.
Connecting to VectorAI DB at localhost:6574 ...
Building VectorStoreIndex (embedding + inserting into VectorAI DB) ...
Ingestion complete.
Collection 'document_intelligence': 312 points in VectorAI DB.

The collection is written to the Docker volume defined in docker-compose.yml. It survives restarts.

Build the Query Agent

Wire the index to a local LLM and build the query interface. The agent.py script uses the external client pattern: a VectorAIClient context manager wraps the entire query session, and ActianVectorAIVectorStore receives it as the client argument. This gives you explicit control over the connection lifecycle when the agent processes multiple queries.

# agent.py -- QueryEngine setup and first query with cited output

import argparse
import sys
from typing import Any

from actian_vectorai import VectorAIClient
from llama_index.core import VectorStoreIndex, StorageContext, Settings
from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama

from vectoraidb_vectorstore import ActianVectorAIVectorStore

VECTORAIDB_URL  = "localhost:6574"
COLLECTION_NAME = "document_intelligence"
EMBED_MODEL     = "nomic-embed-text"
LLM_MODEL       = "llama3.2"
TOP_K           = 5


def parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="Query the document intelligence agent")
    p.add_argument("--query",      default=None,            help="Single query")
    p.add_argument("--react",      action="store_true",     help="Use ReActAgent")
    p.add_argument("--collection", default=COLLECTION_NAME, help="Collection name")
    p.add_argument("--url",        default=VECTORAIDB_URL,  help="VectorAI DB host:port")
    p.add_argument("--top-k",      type=int, default=TOP_K, help="Chunks to retrieve")
    return p.parse_args()


def configure_settings() -> None:
    Settings.embed_model = OllamaEmbedding(
        model_name=EMBED_MODEL,
        base_url="http://localhost:11434",
    )
    Settings.llm = Ollama(
        model=LLM_MODEL,
        base_url="http://localhost:11434",
        temperature=0,       # temperature 0 for factual retrieval tasks
        request_timeout=120.0,
    )


def print_response(response: Any) -> None:
    print("\n" + "=" * 60)
    print("ANSWER:")
    print(response.response if hasattr(response, "response") else str(response))
    if hasattr(response, "source_nodes") and response.source_nodes:
        print("\nSOURCES:")
        for i, node in enumerate(response.source_nodes, 1):
            meta = node.node.metadata
            fname = meta.get("file_name", meta.get("source", "unknown"))
            chunk = meta.get("chunk_index", "?")
            score = f"{node.score:.4f}" if node.score is not None else "n/a"
            print(f"  [{i}] {fname} | chunk {chunk} | score {score}")
    print("=" * 60 + "\n")


def run_query_engine(index: VectorStoreIndex, query: str, top_k: int) -> None:
    engine = index.as_query_engine(
        similarity_top_k=top_k,
        response_mode="compact",
    )
    print(f"\nQuery: {query}")
    response = engine.query(query)
    print_response(response)


def interactive_loop(index: VectorStoreIndex, react: bool, top_k: int) -> None:
    print("Entering interactive mode. Type 'exit' or 'quit' to end.")
    while True:
        try:
            query = input("Query> ").strip()
        except (EOFError, KeyboardInterrupt):
            print("\nExiting interactive mode.")
            break


        if not query or query.lower() in {"exit", "quit"}:
            print("Exiting interactive mode.")
            break


        if react:
            print("ReActAgent mode selected; using the query engine fallback.")


        run_query_engine(index, query, top_k)

def main() -> None:
    args = parse_args()
    configure_settings()

    print(f"Connecting to VectorAI DB at {args.url} ...")

    # External client pattern: pass a pre-configured VectorAIClient.
    # url and client_kwargs are ignored when client= is provided.
    # The caller (this script) is responsible for the client lifecycle.
    with VectorAIClient(args.url) as client:
        try:
            info = client.get_collection(args.collection)
            print(f"Collection '{args.collection}' loaded: {info.points_count} points.")
        except Exception:
            print(f"\nERROR: Collection '{args.collection}' not found.")
            print("Fix: run 'uv run ingest.py' first.")
            sys.exit(1)

        vector_store = ActianVectorAIVectorStore(
            client=client,
            collection_name=args.collection,
            stores_text=True,
        )
        storage_context = StorageContext.from_defaults(vector_store=vector_store)

        # Load from existing collection -- no re-ingestion.
        # This is the persistence proof: the same collection is intact
        # from the previous session without any rebuild step.
        index = VectorStoreIndex.from_vector_store(
            vector_store,
            storage_context=storage_context,
        )
        print("Index ready. Knowledge base is intact from the previous session.\n")

        if args.query:
            run_query_engine(index, args.query, args.top_k)
        else:
            interactive_loop(index, args.react, args.top_k)


if __name__ == "__main__":
    main()

Run a single query:

uv run agent.py --query "What are the main data retention policies described across all documents?"

Expected output:

Connecting to VectorAI DB at localhost:6574 ...
Collection 'document_intelligence' loaded: 312 points.
Index ready. Knowledge base is intact from the previous session.

Query: What are the main data retention policies described across all documents?

============================================================
ANSWER:
According to the documents, data retention policies vary by record type.
The compliance handbook specifies a 7-year retention period for financial
records. The HR policy requires personnel files to be retained for the
duration of employment plus 5 years. The IT security policy mandates
that access logs be kept for 90 days.

SOURCES:
  [1] compliance_handbook.pdf | chunk 14 | score 0.8821
  [2] hr_policy_v3.pdf | chunk 7 | score 0.8643
  [3] it_security_policy.pdf | chunk 22 | score 0.8417
  [4] compliance_handbook.pdf | chunk 15 | score 0.8201
  [5] hr_policy_v3.pdf | chunk 8 | score 0.8089
============================================================

The agent retrieves from multiple PDFs in a single query and cites the source file and chunk index for each result.

Convert the QueryEngine to a ReActAgent for multi-step reasoning.

# ReActAgent setup from the existing index
def run_react_agent(index: VectorStoreIndex, query: str, top_k: int) -> None:
    engine = index.as_query_engine(similarity_top_k=top_k)

    # Wrap the QueryEngine as a tool the ReActAgent can call repeatedly
    tool = QueryEngineTool(
        query_engine=engine,
        metadata=ToolMetadata(
            name="document_knowledge_base",
            description=(
                "Use this tool to retrieve information from the indexed document library. "
                "Input should be a precise natural-language question."
            ),
        ),
    )

    # Build the ReActAgent from the tool
    agent = ReActAgent.from_tools(
        tools=[tool],
        llm=Settings.llm,
        verbose=True,
        max_iterations=10,
    )

    print(f"\nReActAgent query: {query}")
    response = agent.chat(query)
    print_response(response)

Run it:

uv run agent.py --react --query "Compare the data retention period for financial records with the retention period for access logs. Which is longer, and by how much?"

The ReActAgent plans a multi-step retrieval strategy, issues separate queries against the tool for each record type, and synthesizes the results into a single answer. Because the VectorAI DB collection is durable, the agent starts each session with the full indexed document library intact. It does not rediscover what it already knows.

Persistence proof.

Stop the agent process. Restart it. The VectorStoreIndex.from_vector_store call in main() reconnects to the same collection and returns the same point count without any re-ingestion step. Run:

uv run agent.py --query "Summarize the key policies in these documents"

The collection is intact from the previous session. The agent picks up exactly where it left off.

Interactive mode.

Omit -- query to drop into a prompt loop:

uv run agent.py          # QueryEngine interactive
uv run agent.py --react  # ReActAgent interactive
Interactive mode (QueryEngine). Type 'quit' or 'exit' to stop.

Query> What does the IT security policy say about password rotation?
...
Query> quit

Adapt This to Your Own Document Library

The VectorAI DB collection persists across sessions and survives restarts. An agent built on this stack accumulates knowledge over time. Ingest once, and query across sessions without rebuilding. That property holds regardless of which document library, embedding model, or LLM you use.

Substitution 1: Replace the sample PDFs with any local document collection.

Point SimpleDirectoryReader at a different directory. Remove the required_exts=[".pdf"] filter if your document library is mixed format — SimpleDirectoryReader handles .txt, .md, .docx, and other formats natively. The ingestion pipeline, ActianVectorAIVectorStore setup, and agent pattern are identical whether your document library is an internal knowledge base, legal discovery files, research papers, or support documentation.

Substitution 2: Replace nomic-embed-text with a different embedding model.

If a different embedding dimension is required, pull the new model with Ollama and update EMBED_MODEL. VectorAI DB locks the vector dimension at collection creation, so changing the embedding model means deleting the existing collection and re-ingesting. No code change is needed beyond updating EMBED_MODEL, since ActianVectorAIVectorStore infers the new dimension from the first embedding.

# Example: switch to mxbai-embed-large (1024 dimensions)
Settings.embed_model = OllamaEmbedding(
    model_name="mxbai-embed-large",
    base_url="http://localhost:11434",
)
# No other code change needed -- dimension is inferred on first add()

Substitution 3: Replace llama3.2 with a larger Ollama-compatible model.

Larger models produce better multi-step reasoning at the cost of generation speed. Increase request_timeout for larger models.

# Example: switch to llama3.3:70b
Settings.llm = Ollama(
    model="llama3.3:70b",
    base_url="http://localhost:11434",
    temperature=0,
    request_timeout=300.0,
)

The persistent memory property applies to any of these substitutions. What changes is the document library, the embedding dimension, or the generation quality. The durability of the VectorAI DB collection remains unchanged.

Wrapping Up

You built a local document intelligence agent using LlamaIndex, VectorAI DB, Ollama, and nomic-embed-text.

The solution uses LlamaIndex's custom VectorStore interface to connect VectorAI DB to standard LlamaIndex indexing, retrieval, query engine, and agent workflows. The resulting knowledge base persists across application restarts, allowing your agent to retain access to previously indexed information without re-ingestion.

The GitHub repo accompanying this tutorial includes:

.
├── docker-compose.yml
├── docs/
├── ingest.py
├── agent.py
├── vectoraidb_vectorstore.py
├── pyproject.toml
└── README.md

The ActianVectorAIVectorStore implementation ships as a standalone module that can be imported into any LlamaIndex project.
For deployment guidance and API updates, review the VectorAI DB documentation.

How to Build Persistent Agent Memory Across Sessions

Oluseye Jeremiah — Mon, 13 Jul 2026 14:05:43 +0000

Your agent can follow a rule perfectly at the start of a conversation and still miss it later.

Maybe the rule is simple: don't share user contact details, don't book meetings outside business hours, or don't recommend discontinued products. The agent follows it for the first few turns. Then the conversation gets longer. Tool results pile up. The user changes direction a few times. By turn 15, the original constraint is still technically in the context, but the model is no longer treating it like the most important instruction.

That failure is easy to misread. It is not always a prompt-quality problem, and it is not solved just by using a model with a larger context window. The issue is that the context window was never designed to behave like durable memory.

In this article, we'll build a two-layer memory architecture for agents: one layer for active conversation state and another for facts, preferences, and constraints that need to survive across sessions. The implementation uses Python and Actian VectorAI DB, but the pattern is framework-agnostic and can be implemented in agent frameworks such as LangChain, LangGraph, LlamaIndex, AutoGen, or Mastra.

Why the Context Window Is Not a Database

Most production agent failures come from one mistake: developers treating the context window like a database. The context window is the part of the conversation that's currently influencing the model's next answer. It looks like storage because text accumulates in it, but it behaves more like volatile working memory than durable storage in three ways that matter.

First, the context is volatile. Everything in it is gone when the session ends, no automatic persistence, no recovery. Close the chat, open a new one, and the agent has no memory of what came before.

Position also matters. The model pays more attention to information at the start and end of the context than to information stuck in the middle, a behavior documented in "Lost in the Middle." For agent memory, that translates to a predictable failure: a constraint set in the first turn drifts further from the model's focus as the conversation grows. A 2026 study at the University of Florida measured the drop across 12 models; prohibitions buried in long context were followed 73 percent of the time at turn 5 but only 33 percent by turn 16. Bigger context windows didn't fix it. What matters is where the constraint sits and how much surrounds it.

Cost is the third issue. Every call to the model reprocesses the entire context window. A 50,000-token context costs the same per call whether the last 500 tokens are new or the whole thing is.

The Gamage study tested compliance at multiple turn depths between turns 5 and 16 by setting a prohibition in the first system prompt and then running a controlled conversation in which intervening turns added context on top of it. The researchers tested two kinds of instructions: omissions (things the agent should not do, such as "never reveal credentials") and commissions (things the agent should always do, such as "always cite sources"). The omissions decayed across turns while commissions held steady at 100 percent.

Figure 1. Commission constraints hold; omission constraints decay as conversation grows.

The asymmetry is the core issue. The model isn't getting worse overall. Prohibitions, specifically the rules that protect users, enforce policy, and prevent leaks, fade as conversation grows. A reliable memory design has to keep those rules close to the model's active instructions.
Once the context window is treated as temporary working memory, the next question is what kinds of information should exist outside it.

The Four Types of Agent Memory

Cognitive Architectures for Language Agents (CoALA), a 2023 framework from Princeton, provides a useful taxonomy widely referenced in modern agent-memory discussions. It identifies four types of agent memory.
Working memory is the active context window. It holds the current task, recent tool results, the ongoing conversation, and the current task state. It gets cleared at session end.

Episodic memory is a record of specific past events what happened in previous sessions, what tools were called and what they returned, what the user said three conversations ago. It has to be stored externally to survive session resets.

Semantic memory is structured factual knowledge: user preferences, domain facts, and entity relationships. It doesn't change often and lives externally.

Procedural memory is rules, workflows, and decision patterns, usually implemented as part of the system prompt or as retrievable tool definitions.

In practice, episodic and semantic memory live in the same external storage layer. The distinction matters for retrieval logic, not for storage infrastructure. That gives us the design boundary for the rest of the article: working memory stays in the prompt, while durable facts and past events move into external storage.

The Two-Layer Architecture

The design becomes much easier if you separate memory by lifespan. Information that needs to persist into a future session goes into persistent storage; anything that only matters for the current session stays in the context window.

The working memory layer (the context window) holds the current task and immediate user request, tool results summarized to relevant facts before injection rather than raw API responses, the current plan and task state, and hard constraints re-injected at the top of every system prompt call. Constraints do not live in conversation history. They sit at the top of every system prompt, so the model can't bury them.

The persistent memory layer (external storage) holds user preferences with exact values rather than summaries: "user prefers 2700K warm white lighting after 8 p.m." rather than "user has lighting preferences." It also holds hard constraints that must survive session boundaries, identity facts like timezone and project context, and task state that needs to survive a session reset.

Figure 2. The two-layer memory architecture. Working memory clears at the end of the session; persistent memory survives.

At the end of every session, run an extraction pass that writes stable preferences and facts to persistent storage and discards everything else. That extraction is the primary route for transferring information from working memory to persistent memory.

Before writing anything to that external layer, though, it's worth deciding what should never be stored in the first place.

Memory Safety: What Not to Persist

Persistent memory is sensitive and should be treated like any other store of user data, which means writing only the stable facts the agent will need later rather than everything the user says. Tag each memory by type preference, constraint, identity fact, task state — so retrieval logic and access controls can handle them differently.

Hard constraints in particular should not be mixed with casual preferences. A policy that says "never share medical records" is not the same as "user prefers email over Slack," and storing them in the same untagged bucket invites confusion later. Apply standard data hygiene: minimize PII, give sensitive fields a retention window and a deletion path, encrypt at rest, and put access controls in front of the store. Consent for what gets remembered should match the surface where the agent is deployed.

It’s important to also plan for stale and conflicting memories. A user who preferred email in session 1 might switch to Slack in session 5. If both records exist and there is no metadata to choose between them, retrieval can return contradictory facts, and the agent will pick one at random. Stamp every memory with a timestamp and a memory type, and when an extractor writes a new preference, supersede the older record rather than storing both. Retrieval logic can then prefer the most recent valid fact for any given type.

With those rules in place, the storage layer becomes easier to choose. The right backend depends on where the memory is allowed to live, how much retrieval volume you expect, and how much infrastructure the team already operates.

Building the Persistent Memory Layer

Three paths cover most cases.
Teams already running Postgres with pgvector, Actian Zen, or HCL Informix® can embed vector search directly in their existing database. If the memory layer fits comfortably within that system, there's no reason to add another one. One system to operate, one place to back up.

A dedicated vector store is the right fit when the memory layer must stay local or on-premises, when the workload calls for filter-heavy retrieval, or when the dataset will outgrow what a relational extension comfortably handles. This is the most explicit version of the architecture, since the vector store exists for one job: serving the memory layer.

For deduplication, multi-strategy retrieval, and managed extraction pipelines, use a purpose-built memory layer like Mem0 or Zep.

For this tutorial, we'll use the dedicated vector store path because it shows the memory pattern most clearly, and we'll use VectorAI DB as the implementation. It runs locally and on-premises with Python and JavaScript clients designed for agent workloads. The same pattern works with a relational vector extension or a managed memory platform; only the client changes.

The implementation has four moving parts: start the vector store, create a memory collection, write extracted facts at session close, and retrieve relevant facts at the next session start. The agent loop at the end ties them together.

Run the vector store
VectorAI DB runs as a Docker container. The image exposes gRPC on port 6574 and a local web UI on 6575:

docker pull actian/vectorai:latest
docker run -d --name vectorai \
  -v ./local_data:/var/lib/actian-vectorai \
  -p 6573-6575:6573-6575 \
  -e ACTIAN_VECTORAI_ACCEPT_EULA=YES \
  actian/vectorai:latest

Figure 3. VectorAI DB running locally in a Docker container.

Install the Python client:

pip install actian-vectorai-client sentence-transformers

Create the memory collection

The memory record carries the verbatim text in the payload's content field so retrieval returns exact values, not summaries. Agent ID, session ID, and timestamp let you filter retrieval by agent, conversation, or recency:

from actian_vectorai import VectorAIClient, VectorParams, Distance
from sentence_transformers import SentenceTransformer
EMBED_MODEL = "all-MiniLM-L6-v2"
EMBED_DIM = 384
COLLECTION = "agent_memory"
embedder = SentenceTransformer(EMBED_MODEL)

with VectorAIClient("localhost:6574") as client:
    client.collections.get_or_create(
        name=COLLECTION,
        vectors_config=VectorParams(size=EMBED_DIM, distance=Distance.Cosine),
    )

Figure 4. The agent_memory collection in the VectorAI DB local UI.

Write at session close

At the end of every session, extract the stable facts from the conversation and write them down. Each memory is one PointStruct: an integer ID, an embedding of the content, and a payload that carries the verbatim text plus the routing metadata.

from actian_vectorai import VectorAIClient, PointStruct
from datetime import datetime, timezone
import uuid

def write_memory(content: str, agent_id: str, session_id: str):
    """Persist one extracted memory at session close."""
    vector = embedder.encode(content).tolist()
    memory_id = uuid.uuid4().int >> 64  # 64-bit positive int

    with VectorAIClient("localhost:6574") as client:
        client.points.upsert(COLLECTION, points=[
            PointStruct(
                id=memory_id,
                vector=vector,
                payload={
                    "content": content,                      
                    "agent_id": agent_id,
                    "session_id": session_id,
                    "timestamp": datetime.now(timezone.utc).isoformat(),
                },
            ),
        ])
        client.vde.flush(COLLECTION)

Retrieve at session start

At the start of the session, retrieve memories that match the user's likely intent. Filter by agent so memories don't leak between agents that share the store:

from actian_vectorai import VectorAIClient, Field, FilterBuilder
def retrieve_memories(query: str, agent_id: str, k: int = 5):
    """Pull the k most relevant memories for this agent."""
    query_vector = embedder.encode(query).tolist()
    agent_filter = FilterBuilder().must(Field("agent_id").eq(agent_id)).build()

    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
            filter=agent_filter,
        ) or []

Figure 5. Semantic retrieval returning the most relevant memory by vector similarity, not keyword match.

The agent loop

A simple baseline is one read at session start and one write at session close. Longer sessions, agents that take consequential actions, or workflows that may exit unexpectedly, need checkpoint writes after important state changes. That might be after a confirmed preference, after a tool call that mutates external state, or before a risky action. The pattern below is a starting point that production systems typically extend.

def run_agent_session(user_id: str, agent_id: str, hard_constraints: list[str]):
    session_id = str(uuid.uuid4())

    # 1. Read at session start
    memories = retrieve_memories(query=f"context for user {user_id}", agent_id=agent_id)
    memory_text = "\n".join(m.payload["content"] for m in memories)

    # 2. Build the system prompt with constraints pinned at the top, every call
    system_prompt = (
        "CONSTRAINTS (always apply):\n"
        + "\n".join(f"- {c}" for c in hard_constraints)
        + "\n\nRELEVANT MEMORY:\n"
        + memory_text
    )

    # 3. Run the conversation. Re-prepend the constraint block on every LLM call,
    #    not just at session start. Summarize tool results before injecting them.
    conversation = run_conversation(system_prompt)

    # 4. Write at session close: extract stable facts and persist
    for fact in extract_stable_facts(conversation):
        write_memory(content=fact, agent_id=agent_id, session_id=session_id)

run_conversation and extract_stable_facts are placeholders for your agent runtime and your extraction logic. A reasonable implementation of extract_stable_facts uses the LLM with a focused prompt to identify durable preferences and facts from the transcript ("user mentioned they prefer email over Slack" graduates to persistent memory; "user said hi" does not). Keep the extractor narrow, because anything it writes will influence future sessions.

Figure 6. The agent loop: one read at session start, one write at session close.

Once the read and write hooks are in place, the next step is proving that the memory layer actually fixes the failure modes the article started with.

Testing That It Works

Four tests will tell you whether the architecture is ready for a pilot. Each one targets a specific failure mode and provides a specific fix if it doesn't pass.

Constraint adherence by turn depth. Set one constraint at turn 1, then run the agent through turn 15. Check compliance at turns 5, 10, and 15. Target: above 90 percent at all three. If it drops below 70 percent by turn 10, the constraint probably isn't being pinned correctly. Pull it from persistent memory at session start and prepend it to the system prompt on every LLM call, not just the first one.

Exact value accuracy. Store a specific preference with an exact value ("2700K warm white lighting after 8 p.m."). Retrieve it two sessions later without restating it. Check that the exact value comes back, not a paraphrase.

Fix: Store the verbatim string in the content field of the memory record. Semantic search retrieves what's in the content, so if the exact value is there, it comes back.

Token growth over time. Measure working-memory token count at the start and end of a 20-turn session. A flat line or gentle growth indicates healthy management. Exponential growth means tool results are being injected raw.

Fix: Add a summarization step between tool execution and context injection. Inject only the facts the agent needs to continue.

Cross-session recall. State a fact in session 1, close the agent, open session 2, ask a question that requires that fact. If the agent asks for the fact again, the session-close extraction isn't running.

Fix: Confirm the extraction function is being called at session end (including non-graceful exits) and that client.vde.flush() completes before the session terminates.

If all four pass, the memory layer is ready for a basic production pilot. Full production readiness also depends on access control, encryption, observability, retention policy, and failure handling, none of which are covered by these tests.

Figure 7. Cross-session recall in practice: identical retrieval scores after a full container restart confirm the memories persisted to disk and were not rebuilt.

Wrapping Up

Persistent agent memory starts with deciding what belongs in the prompt and what belongs in storage. The context window holds the current task, recent facts, and summarized tool outputs. Anything that must survive the session's hard constraints, exact user preferences, or task state lives in persistent storage and is retrieved deliberately.
That separation is what prevents the turn-15 failure from the opening. The agent no longer depends on an outdated instruction buried in the conversation history.
For deeper implementation detail, see the VectorAI DB Python SDK reference for collection state, payload indexes, and connection pooling, and the legal contract intelligence agent tutorial for a more advanced example.

Informix is a trademark of IBM Corporation in at least one jurisdiction and is used under license.

Multi-Collection Vector Search for Clinical Data

Tiioluwani — Mon, 13 Jul 2026 13:32:56 +0000

You are testing your clinical search system and ask it to find patients with lab profiles similar to a reference case for chronic kidney disease. The results come back with full discharge summaries and progress notes that mention creatinine only in passing. But not a single lab record appears in the top ten. This happens because of how the system is built.

If you use just one embedding model trained on general medical text, it cannot distinguish between a clinical note that says “elevated creatinine” and a lab record showing a creatinine level of 4.2 mg/dL. In a vector space, where data is turned into arrays of numbers, both end up close together because they use similar words. But one is a doctor’s note, and the other is an actual measurement, and your search system can’t tell them apart.

This article builds a multi-collection vector search system for clinical data using VectorAI DB, FastAPI, and Docker. By the end, you will have six collections, one per clinical data type, queried in parallel through a single FastAPI gateway that runs entirely on-premises, with no patient data leaving your infrastructure

Why Clinical Data Breaks Single-Collection Search

When teams build a vector search system for clinical data, they often start with a single collection and a single embedding model. In an EHR system, though, different data types have different structures, vocabularies, and search patterns.

Data type	Example record	What breaks in a shared collection
Clinical notes	Discharge summary: "Patient presented with chest pain radiating to the left arm. EKG showed ST elevation."	Notes dominate search results because they contain the most natural language. Other data types are buried.
Imaging records	Radiology report metadata: modality CT, body region thorax, finding "2.3 cm pulmonary nodule", associated image stored in a Picture Archiving and Communication System (PACS)	Imaging metadata fields such as modality and Digital Imaging and Communications in Medicine (DICOM) Study UID do not belong in a schema designed for free-text notes.
Lab results	Creatinine: 4.2 mg/dL, reference range 0.7 to 1.3, collected 2027-01-15, ordering provider: nephrology	Numeric values with units are poorly suited to text embedding models. A creatinine result of 4.2 mg/dL and one of 0.8 mg/dL may land close together in the embedding space because the surrounding text looks similar, even though clinically they are very different.
Medication records	Metformin 500mg, twice daily, start date 2025-03-01, prescriber: endocrinology, RxNorm (a standardized vocabulary for medications) code 860974	Drug names share vocabulary with clinical notes and discharge summaries. Medication searches return narrative mentions instead of actual medication history.
Structured EHR	International Classification of Diseases, 10th revision (ICD-10) code E11.9 (Type 2 diabetes without complications), encounter date 2026-04-15, visit type: outpatient	Coded data in ICD-10 and SNOMED carry meaning only within their own vocabulary systems. General-purpose embedding models lack a reliable way to represent that structure, so coded records yield poor similarity results when mixed with free text.
Knowledge base	PubMed abstract on chronic kidney disease management, institutional guideline on antibiotic prescribing, clinical reference entry on drug-drug interactions	Without its own collection, clinical reference material mixes with patient records, making it impossible to separate evidence-based citations from patient-specific data at query time.

If you put all six data types into one collection, your embedding models end up handling tasks they were not built for. Clinical notes take over because they have the most natural language. Lab values, medication records, and coded diagnoses get lost because numbers and coding systems like ICD-10 and SNOMED do not integrate well with unstructured text. As a result, search accuracy drops for every data type except the one that best matches the model.

Lee et al. showed this in npj Digital Medicine. In a study of over 400,000 emergency department visits, they found that embedding EHR data domains separately before combining them performed better than using a single mixed-embedding model for every clinical prediction task. If you combine everything into one set of vector embeddings, you lose the unique value of each data type. Using a better embedding model will not fix this, but separating your collections will.

Multi-Collection Search

With multi-collection vector search, each clinical data type has its own separate collection and embedding model. When you send a search query, it is turned into a query vector and sent to all the relevant collections at once. Each collection searches independently using Approximate Nearest Neighbor (ANN) algorithms, which quickly narrow down results from millions of records.

This architecture splits clinical data across six collections, each designed for a specific data type:

clinical_notes: discharge summaries, progress notes, and consult notes
imaging_records: radiology and pathology report metadata; image files stay in a separate PACS system
lab_results: numeric lab values, reference ranges, and Logical Observation Identifiers Names and Codes (LOINC) codes
medications: medication names, doses, durations, and RxNorm codes
ehr_structured: coded diagnosis and procedure records using ICD-10 and SNOMED (Systematized Nomenclature of Medicine Clinical Terms) codes
knowledge_base: clinical reference content from PubMed and institutional guidelines

Since each collection has its own embedding model, you cannot directly compare similarity scores between collections. To solve this, the system merges results using Reciprocal Rank Fusion (RRF). RRF ranks items based on their positions in each result list, not on their raw scores, and produces a single combined ranking.

But before you deploy this architecture in a clinical environment, there is a regulatory layer you need to understand.

The On-Premises Compliance Requirement

HIPAA, the Health Insurance Portability and Accountability Act, is the United States federal law that governs the privacy and security of health data. Any data that can identify a patient and relates to their health condition, treatment, or payment is Protected Health Information (PHI), including names, dates of birth, diagnoses, lab results, and prescriptions.

In this architecture, the vector database, embedding service, and API gateway all store, process, or transmit PHI, which means each component falls under HIPAA's Security Rule. Three requirements affect this architecture directly:

Data residency: PHI cannot be sent to a cloud service without a Business Associate Agreement (BAA), a contract that makes the vendor legally responsible for protecting PHI under HIPAA. Sending PHI to a cloud-hosted vector database without a BAA in place is a HIPAA violation.
Encryption: Under the current HIPAA Security Rule, encryption of PHI at rest is an addressable specification, meaning covered entities must assess whether it is reasonable and appropriate for their environment. The 2025 Notice of Proposed Rulemaking (NPRM) proposes making encryption mandatory for all systems that access PHI. As of this writing, that requirement is still proposed.
Audit logging: Under 45 CFR section 164.312(b), all PHI access must be logged. Under 45 CFR section 164.316(b)(2)(i), these logs must be retained for six years.

VectorAI DB runs on-premises fully using Docker, so PHI always stays within your own infrastructure. The database uses AES-256 encryption for data at rest and lets you manage your own encryption keys. This removes the need for a BAA at the vector database layer itself. Any external services or managed infrastructure that handles PHI elsewhere in your stack will still require a BAA and a compliance review.

Check with your organization’s compliance and legal teams before making any architectural decisions based on these requirements.

The Full Stack

" width="799" height="313">

The system has five components:

VectorAI DB: Stores all six clinical collections on your own servers and handles similarity search using Hierarchical Navigable Small World (HNSW) indexing, a graph-based structure that makes ANN search fast at scale
FastAPI: Receives search requests from clinical applications, validates them, checks caller permissions, and forwards the query gRPC: The communication protocol between FastAPI and VectorAI DB; uses a binary format for internal service communication
Docker: Packages VectorAI DB and the FastAPI service into isolated, reproducible containers that run consistently on any server
Embedding service: Converts incoming queries and clinical records into vector embeddings locally: no external API calls, no PHI leaves the server

The recommended hardware is Ubuntu 22.04 LTS, 16 GB of RAM, an 8-core CPU, and a 200 GB NVMe SSD. A GPU can speed up embedding generation, but it is not needed for search.

Note: This tutorial covers only dense vector search. Adding hybrid search, which mixes dense ANN search with keyword matching, can improve recall for queries with specific drug names, LOINC codes, or ICD-10 codes. Once your base system works, this is the next recommended step. Check out the VectorAI DB documentation for details on hybrid search.

Setting Up the Collections

Before you begin, make sure Docker Desktop is installed. Install Python 3.10 or later, and confirm that pip is available.

Project structure

Your project will have the following structure, which keeps the collection setup, ingest, search, and audit steps separated by responsibility:

clinical-search/
├── docker-compose.yml
└── src/
    ├── clinical_collections.py
    ├── ingest.py
    ├── search.py
    ├── audit.py
    ├── evaluate.py
    └── main.py

Stage 1: Start VectorAI DB with Docker

Create a docker-compose.yml file in your project root:

services:
  vectorai:
    image: actian/vectorai:latest
    container_name: vectorai
    ports:
      - "6573:6573"
      - "6574:6574"
      - "6575:6575"
    volumes:
      - ./local_data:/var/lib/actian-vectorai
    environment:
      - ACTIAN_VECTORAI_ACCEPT_EULA=YES
    restart: unless-stopped

Start the container:

docker compose up -d

Confirm it is running:

docker ps --filter name=vectorai --format 'Status: {{.Status}}  Ports: {{.Ports}}'

Open http://localhost:6575 in your browser. A System Health of 100% confirms that the database is ready.

Stage 2: Connect with VectorAIClient

Install the Python SDK:

pip install actian-vectorai-client

Verify the connection:

from actian_vectorai import VectorAIClient

with VectorAIClient("localhost:6574") as client:
    info = client.health_check()
    print(f"Connected: {info}")

Expected output:

Connected: {'title': 'Actian VectorAI DB', 'version': 'Actian VectorAI DB 1.0.1 / VDE 1.0.1'}

Stage 3: Create the six collections

Each collection uses 768-dimensional vectors with cosine similarity, includes a primary key field, and carries five compliance metadata fields: patient_id, site_id, encounter_id, data_sensitivity_class, and created_at.

These fields enable HIPAA-compliant filtering and audit logging across every collection. This dense-only schema is enough for storing dense vectors today. If you later extend to multi-vector hybrid search, it can also support additional vector fields, raw text descriptions, and sparse-vector fields for full-text search.

Set up src/clinical_collections.py to define and initialize all six collections:

from actian_vectorai import VectorAIClient
from actian_vectorai.models import VectorParams, Distance

HOST = "localhost:6574"

COLLECTIONS = [
    {"name": "clinical_notes", "size": 768},
    {"name": "imaging_records", "size": 768},
    {"name": "lab_results", "size": 768},
    {"name": "medications", "size": 768},
    {"name": "ehr_structured", "size": 768},
    {"name": "knowledge_base", "size": 768},
]

def create_collections():
    with VectorAIClient(HOST) as client:
        for col in COLLECTIONS:
            try:
                client.collections.create(
                    col["name"],
                    vectors_config=VectorParams(
                        size=col["size"],
                        distance=Distance.Cosine,
                    ),
                )
                print(f"Created: {col['name']}")
            except Exception as e:
                print(f"Skipped {col['name']}: {e}")

if __name__ == "__main__":
    create_collections()

All six collections are now visible in the Local UI under Collections.

Stage 4: Ingest a sample clinical record

The embedding service generates vector embeddings from each record's text before insertion. In production, you should use your own clinical embedding model instead of get_mock_embedding. You can also use precomputed dense vectors if embeddings are produced upstream. The client.points.upsert method adds each record to its collection, creating a new entry if one does not exist or updating it if one does.

Add src/ingest.py with a mock embedding function and a synthetic record for each collection.

from actian_vectorai import VectorAIClient, PointStruct
import hashlib

HOST = "localhost:6574"

def get_mock_embedding(text: str, size: int = 768) -> list[float]:
    hash_bytes = hashlib.sha256(text.encode()).digest()
    return [((hash_bytes[i % len(hash_bytes)] / 255.0) * 2 - 1) for i in range(size)]

SYNTHETIC_RECORDS = [
    {
        "collection": "clinical_notes",
        "id": 1,
        "text": "Patient TEST-001 presented with elevated creatinine levels and reduced urine output. Assessment consistent with acute kidney injury stage 2. Recommended nephrology consult and IV fluid resuscitation.",
        "payload": {
            "patient_id": "TEST-001",
            "site_id": "DEMO-SITE",
            "encounter_id": "ENC-2027-001",
            "data_sensitivity_class": "PHI",
            "created_at": "2027-01-15T09:30:00Z",
            "note_type": "progress_note",
            "author": "DR-DEMO-001",
        }
    },
    {
        "collection": "lab_results",
        "id": 2,
        "text": "Creatinine 4.2 mg/dL reference range 0.7-1.3 mg/dL LOINC 2160-0 status abnormal critical",
        "payload": {
            "patient_id": "TEST-001",
            "site_id": "DEMO-SITE",
            "encounter_id": "ENC-2027-001",
            "data_sensitivity_class": "PHI",
            "created_at": "2027-01-15T08:00:00Z",
            "loinc_code": "2160-0",
            "value": 4.2,
            "unit": "mg/dL",
            "reference_range": "0.7-1.3",
            "status": "critical",
        }
    },
    {
        "collection": "medications",
        "id": 3,
        "text": "Metformin 500mg twice daily oral type 2 diabetes management RxNorm 860974",
        "payload": {
            "patient_id": "TEST-001",
            "site_id": "DEMO-SITE",
            "encounter_id": "ENC-2027-001",
            "data_sensitivity_class": "PHI",
            "created_at": "2027-01-15T09:00:00Z",
            "rxnorm_code": "860974",
            "dose": "500mg",
            "frequency": "twice daily",
            "route": "oral",
        }
    },
    {
        "collection": "ehr_structured",
        "id": 4,
        "text": "ICD-10 E11.9 type 2 diabetes without complications outpatient encounter",
        "payload": {
            "patient_id": "TEST-001",
            "site_id": "DEMO-SITE",
            "encounter_id": "ENC-2027-001",
            "data_sensitivity_class": "PHI",
            "created_at": "2027-01-15T09:00:00Z",
            "icd10_code": "E11.9",
            "description": "Type 2 diabetes mellitus without complications",
            "encounter_type": "outpatient",
        }
    },
    {
        "collection": "imaging_records",
        "id": 5,
        "text": "CT abdomen pelvis contrast enhanced bilateral renal cortical thinning consistent with chronic kidney disease no acute obstruction",
        "payload": {
            "patient_id": "TEST-001",
            "site_id": "DEMO-SITE",
            "encounter_id": "ENC-2027-001",
            "data_sensitivity_class": "PHI",
            "created_at": "2027-01-15T10:00:00Z",
            "modality": "CT",
            "body_region": "abdomen_pelvis",
            "finding": "bilateral renal cortical thinning",
            "pacs_reference": "PACS-DEMO-001",
        }
    },
    {
        "collection": "knowledge_base",
        "id": 6,
        "text": "Acute kidney injury staging KDIGO criteria stage 2 creatinine 2-2.9x baseline or urine output less than 0.5 mL/kg/h for 12 hours management includes nephrology consult fluid resuscitation and nephrotoxin avoidance",
        "payload": {
            "patient_id": "NONE",
            "site_id": "DEMO-SITE",
            "encounter_id": "NONE",
            "data_sensitivity_class": "PUBLIC",
            "created_at": "2027-01-01T00:00:00Z",
            "source": "KDIGO_Guidelines_2024",
            "topic": "acute_kidney_injury",
        }
    },
]

def ingest_records():
    with VectorAIClient(HOST) as client:
        for record in SYNTHETIC_RECORDS:
            vector = get_mock_embedding(record["text"])
            point = PointStruct(
                id=record["id"],
                vector=vector,
                payload=record["payload"],
            )
            client.points.upsert(record["collection"], [point])
            print(f"Ingested record {record['id']} into {record['collection']}")

if __name__ == "__main__":
    ingest_records()
    print("All synthetic records ingested successfully.")

Open the clinical_notes collection in the Local UI to confirm the record was ingested and the metadata fields are visible.

" width="800" height="384">

Note: When you move to production, replace get_mock_embedding with a domain-specific model suited to each collection. ClinicalBERT works well for clinical_notes and imaging_records, PubMedBERT for lab_results and knowledge_base, and SapBERT for medications and ehr_structured. Each model is available via Hugging Face Transformers.

Building the Search Gateway

The FastAPI gateway handles all clinical search requests. When a query comes in, the gateway checks it, embeds the query text, sends the resulting vector to each target collection via gRPC, and then merges the results with RRF before returning a ranked response.

Each collection search call accepts only one query data item, so separate search requests are needed for each target collection.

Install the dependencies:

pip install fastapi uvicorn pydantic

Start with the function that searches all requested collections in turn. It converts the incoming query into a vector embedding, runs the search against each collection sequentially, and merges the results using RRF.

Add src/search.py with the multi-collection search function and the result formatter:

from actian_vectorai import VectorAIClient, reciprocal_rank_fusion, Field, FilterBuilder
from ingest import get_mock_embedding

HOST = "localhost:6574"

def search_collections(query: str, collections: list[str], limit: int = 5, site_id: str = None):
    query_vector = get_mock_embedding(query)
    all_results = []

    with VectorAIClient(HOST) as client:
        for collection in collections:
            filter_ = None
            if site_id:
                filter_ = (
                    FilterBuilder()
                    .must(Field("site_id").eq(site_id))
                    .build()
                )

            results = client.points.search(
                collection,
                vector=query_vector,
                limit=limit,
                filter=filter_,
                with_payload=True,
            )
            all_results.append(results)

    # RRF ranks results by position across lists, not raw score.
    # This is necessary because scores from different embedding
    # models are not directly comparable across collections.
    merged = reciprocal_rank_fusion(all_results, limit=limit)
    return merged

def format_results(results):
    formatted = []
    for i, result in enumerate(results):
        formatted.append({
            "rank": i + 1,
            "id": result.id,
            "score": round(result.score, 4) if result.score is not None else 0.0,
            "payload": result.payload,
        })
    return formatted

The format_results function normalizes the merged RRF output into a consistent structure, assigning each result a rank, ID, score, and payload regardless of which collection it came from.

Within each collection, vector distance is computed locally, so raw scores are not compared across collections; instead, RRF is used.

With the search function in place, wire it into the FastAPI endpoint. The endpoint receives incoming search requests, validates them using Pydantic, a Python library that checks that request data has the correct structure before the code processes it, and triggers an audit log entry on every query.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from search import search_collections, format_results
from audit import emit_audit_event
import uuid

app = FastAPI(title="Clinical Search Gateway")

class SearchRequest(BaseModel):
    query: str
    collections: list[str]
    limit: int = 5
    site_id: str = "DEMO-SITE"
    principal_id: str = "USER-DEMO-001"

class SearchResponse(BaseModel):
    request_id: str
    query: str
    results: list[dict]
    collections_queried: list[str]
    total_results: int

@app.post("/search", response_model=SearchResponse)
def multi_collection_search(request: SearchRequest):
    request_id = str(uuid.uuid4())

    try:
        raw_results = search_collections(
            query=request.query,
            collections=request.collections,
            limit=request.limit,
            site_id=request.site_id,
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

    formatted = format_results(raw_results)

    emit_audit_event(
        principal_id=request.principal_id,
        site_id=request.site_id,
        request_id=request_id,
        collections_queried=request.collections,
        retrieved_record_ids=[r["id"] for r in formatted],
        query=request.query,
    )

    return SearchResponse(
        request_id=request_id,
        query=request.query,
        results=formatted,
        collections_queried=request.collections,
        total_results=len(formatted),
    )

@app.get("/health")
def health():
    return {"status": "ok"}

Start the gateway:
python3 -m uvicorn src.main:app --host 127.0.0.1 --port 8000

Test it with a clinical sample query:

curl -X POST http://127.0.0.1:8000/search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "elevated creatinine acute kidney injury",
    "collections": ["lab_results", "clinical_notes", "knowledge_base"],
    "limit": 3,
    "site_id": "DEMO-SITE",
    "principal_id": "USER-DEMO-001"
  }'

The endpoint returns a ranked JSON response with results from all three collections. It should look like this:

" width="799" height="355">

Audit Logging for HIPAA Compliance

Treat every search query against this system as a PHI access event and log it accordingly. A clinician querying for patients with similar lab profiles is retrieving identifiable health information, and under 45 CFR section 164.312(b), every access must be logged. The query text itself should not appear in the log because it may contain PHI; hash it instead.

Each audit record must capture the principal ID, site ID, request ID, collections queried, retrieved record IDs, a SHA-256 hash of the query text, a timestamp, and the embedding model version. The emit_audit_event function writes each record as a JSON line to an append-only log file after every search.

In src/audit.py, add the function that writes each access event to the log:

import hashlib
import json
from datetime import datetime, timezone

AUDIT_LOG_PATH = "audit.log"

def hash_query(query: str) -> str:
    return hashlib.sha256(query.encode()).hexdigest()[:16]

def emit_audit_event(
    principal_id: str,
    site_id: str,
    request_id: str,
    collections_queried: list[str],
    retrieved_record_ids: list[int],
    query: str,
    model_version: str = "mock-embedder-v1",
):
    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "principal_id": principal_id,
        "site_id": site_id,
        "request_id": request_id,
        "collections_queried": collections_queried,
        "retrieved_record_ids": retrieved_record_ids,
        "query_hash": hash_query(query),
        "model_version": model_version,
    }

    with open(AUDIT_LOG_PATH, "a") as f:
        f.write(json.dumps(record) + "\n")

    return record

Each search event is written to the log with the query hashed and all required fields present:

" width="798" height="67">

Under 45 CFR section 164.316(b)(2)(i), these logs must be retained for six years. Feed the output into your organization's existing Security Information and Event Management (SIEM) system to manage retention. The retention mechanism itself is outside the scope of this tutorial.

Measuring Search Quality

In clinical search, missing a relevant result is more serious than returning an irrelevant one. If you miss a lab record, it could lead to a missed diagnosis. An irrelevant result is just extra noise. These mistakes are not the same, so your quality standards should reflect this difference.

Recall shows what percentage of all relevant records the system found. Precision shows the percentage of returned records that were relevant. For clinical search, Recall is the most important metric.

The evaluation script below calculates Recall@10 and MRR (Mean Reciprocal Rank) against a labeled test set. Replace the synthetic test cases with clinician-labeled queries before running this in production.

Add src/evaluate.py with the Recall@10 and MRR evaluation functions.

def recall_at_k(retrieved_ids: list, expected_ids: list, k: int = 10) -> float:
    retrieved_set = set(retrieved_ids[:k])
    expected_set = set(expected_ids)
    if not expected_set:
        return 0.0
    return len(retrieved_set & expected_set) / len(expected_set)

def mean_reciprocal_rank(retrieved_ids: list, expected_ids: list) -> float:
    expected_set = set(expected_ids)
    for rank, rid in enumerate(retrieved_ids, start=1):
        if rid in expected_set:
            return 1.0 / rank
    return 0.0

The evaluation script prints Recall@10 and MRR scores for each test case, which looks like this:

" width="663" height="510">

Set these goals before launching: Recall@10should be at least 0.95 for safety-critical queries. MRR should be above 0.6, with 0.7 as your target. Citation coverage, which is the percentage of results shown to a clinician that include a traceable source record, should be above 0.9. These thresholds are recommended starting points; you can calibrate them against your own clinician-labeled test set.

Recall@10 of 0.95 or higher is the launch target. Treat 0.90 as the floor: if your clinician-labeled test set falls below 0.90, do not ship. Add a hybrid search that mixes dense vector search with keyword matching, and include a clinical reranker before expanding further.

Wrapping Up

At this point, you have a working multi-collection clinical search system:

Six collections, separated by data type, each with its own embedding model and schema
A FastAPI and gRPC gateway that fans queries out across collections and merges results using RRF
VectorAI DB running on-premises via Docker with HIPAA-aligned configuration and AES-256 encryption at rest

To go deeper into the imaging collection, see the medical image similarity search tutorial.

For collection configuration details and the current API reference, see the VectorAI DB documentation and the Python SDK reference. You can also check out our Community Edition to get started.

Build an OpenClaw Memory Plugin with Actian VectorAI DB

Praise James — Thu, 09 Jul 2026 18:07:12 +0000

Context compaction is a common failure point for most OpenClaw memory systems. When the context window fills, OpenClaw compacts the session to make room for new messages. Any memory operations that were not flushed to durable storage can be lost during this transition.

The default memory-core backend uses a per-agent SQLite database powered by sqlite-vec. This architecture is lightweight, easy to set up, and works well for individual developers running a single agent on a local machine. It becomes fragile when you run multiple agents or when a file-based state needs to be shared or managed outside the runtime.

To address these limitations, this tutorial demonstrates how to build a production-ready OpenClaw memory plugin powered by Actian VectorAI DB. Rather than storing embeddings and memory records inside local SQLite files, the plugin externalizes memory into a dedicated vector database designed for high availability, durability, and independent scaling.

By moving memory management outside the runtime, agents gain access to a centralized knowledge layer that teams can share across processes, hosts, and deployment environments.

How OpenClaw Memory Works by Default

OpenClaw uses a default memory plugin called memory-core. It indexes MEMORY.md into chunks and stores them in a per-agent SQLite database located at ~/.openclaw/memory/<agentId>.sqlite. Inside this file are two key components: an FTS5 (Full-Text Search version 5) index for keyword search and a vec0 vector index powered by the sqlite-vec extension for semantic search.

The important design point is that the Markdown files in the agent workspace are the source of truth. Files like MEMORY.md and memory/YYYY-MM-DD.md hold the actual durable content. The SQLite database does not store primary memory. It only builds a derived, regenerable index over those files. This distinction matters because this tutorial replaces the indexing layer, not the underlying memory files.

In practice, this design introduces some failure modes that push teams toward external memory backends.

Failure modes in the default memory-core backend

Context compaction events can orphan in-flight memory operations. Even though recent fixes in the May 2026 release improve how OpenClaw closes local embedding providers when active-memory searches time out, the underlying issue remains tied to lifecycle boundaries during compaction.

One of the failure modes is that multiple agent instances cannot reliably share a single <agentId>.sqlite file. This creates contention in distributed setups where agents need shared knowledge, leading to inconsistent memory state across instances.

Another failure mode is that sqlite-vec extension must exist in the runtime environment. When it is missing, OpenClaw falls back to in-process JavaScript cosine similarity, which does not scale for production workloads or high-throughput memory search.

Context compaction lifecycle

Context compaction runs when the context window fills and OpenClaw removes older messages to free space. Before compaction begins, OpenClaw triggers an automatic memory flush, but this only captures memories the agent has explicitly written. Any in-flight operations or pending writes that have not reached the memory files are not persisted.

After compaction, the agent resumes with a compressed version of its previous conversation. At that point, memory recall depends on what has already been written to the memory files and indexed for retrieval. Any information that was still in-flight and had not been persisted before compaction cannot be recalled during that session. The issue is not that durable memory disappears, but that memory that was never successfully persisted cannot be recovered.

Because the SQLite database lives inside the agent process, the vector index is tightly coupled to the runtime. When the process restarts or compaction resets the context, the index does not exist independently. Moving the memory layer to VectorAI DB decouples storage from the agent lifecycle, so the memory system survives compaction and continues to serve queries across sessions.

The Plugin Slot System

OpenClaw uses a named slot system to control extensibility points inside the agent runtime. The memory system is one of these slots, exposed as plugins.slots.memory. This slot determines which backend handles all memory-related tool calls.

The slot system separates memory behavior from memory storage. That separation allows you to move from a local SQLite-based index to an external vector database without modifying agent logic or tool definitions. In practice, this means one config change followed by a restart is enough to replace the entire memory backend.

OpenClaw enforces a single active memory plugin at a time. The architecture does not support multiple concurrent memory backends in the same slot. Issue #60572 on GitHub proposes splitting memory into sub-slots such as memory.recall, memory.store, and memory.compaction, but this has not merged as of May 2026. As a result, every implementation must fully replace the existing memory provider rather than extend it.

" width="412" height="512">
Figure 1: Plugin slot architecture

Before writing the VectorAI DB plugin, it helps to study existing implementations of this pattern. The noncelogic/openclaw-memory-lancedb and lancedb/openclaw-lancedb-demo repositories show how external vector stores integrate into the slot system. The serenichron/openclaw-memory-mem0 plugin is the cleanest reference. It demonstrates the minimal structure required to implement a working memory backend and serves as the template for the VectorAI DB integration in this tutorial.

Setting Up VectorAI DB

Before writing the plugin, get VectorAI DB running locally and verify vector search works end-to-end. To execute the commands in this section, install the following:

NPM
- Docker
- Ollama
- OpenClaw CLI
- Together AI API key

In your project directory, create a docker-compose.yml:

services:
 vectorai:
   image: actian/vectorai:latest
   platform: linux/amd64 # If you are using a macOs device, else comment this line.
   container_name: vectorai_db
   ports:
    - "6573:6573"
    - "6574:6574"
   volumes:
    - ./data:/var/lib/actian-vectorai
   environment:
    - VECTORAI_LOG_LEVEL=info
   restart: unless-stopped

Start the VectorAI DB server by:

docker-compose up -d

Ensure Ollama is running and the model is ready:

ollama pull nomic-embed-text
ollama pull llama3.2:3b
ollama serve

Create a package.json file:

{
 "name": "vectorai-memory",
 "version": "1.0.0",
 "type": "module",
 "main": "dist/index.js",
 "openclaw": {
   "extensions": [
     "./dist/index.js"
   ]
 },
 "scripts": {
   "build": "esbuild index.ts --bundle --platform=node --format=esm --external:openclaw --external:@actian/vectorai-client --external:@grpc/grpc-js --external:typebox --outfile=dist/index.js",
   "build:check": "tsc --noEmit"
 },
 "dependencies": {
   "@actian/vectorai-client": "^1.0.2",
   "@sinclair/typebox": "^0.34.49",
   "openclaw": "^2026.5.28",
   "typebox": "^1.1.39"
 },
 "devDependencies": {
   "@types/node": "^22.0.0",
   "esbuild": "^0.28.0",
   "typescript": "^5.0.0"
 }
}

Create a tsconfig.json file:

{
   "compilerOptions": {
       "target": "ES2022",
       "module": "ESNext",
       "moduleResolution": "bundler",
       "strict": true,
       "esModuleInterop": true,
       "skipLibCheck": true
   },
   "include": [
       "index.ts"
   ]
}

Run the command:

npm install

This installs all the dependencies.

Create a collection the plugin will use. In your project directory, create a file create-collection.ts:

import { VectorAIClient } from "@actian/vectorai-client";


const client = new VectorAIClient("localhost:6574", {
   restUrl: "http://localhost:6573",
});


// dim=768 matches nomic-embed-text output dimensions.
await client.collections.create("openclaw-memory", {
   dimension: 768,
   distanceMetric: "COSINE",
});


// Smoke test: insert one vector and query it back.
await client.points.upsert("openclaw-memory", [
   {
       id: 1,
       vector: new Array(768).fill(0.01),
       payload: { text: "smoke test", source: "MEMORY.md", line: 1 },
   },
]);


const results = await client.points.search(
   "openclaw-memory",
   new Array(768).fill(0.01),
   { limit: 1 }
);


console.log("Collection ready. Top result:", results[0].payload.text);

Verify the connection by running the command:

npx tsx create-collection.ts

You will see an output as shown:
" width="512" height="62">
Figure 2: Verify connection

Building the Plugin

The OpenClaw memory slot expects a plugin that implements the memory tools and lifecycle hooks used by the agent runtime. The plugin below replaces the default SQLite/sqlite-vec backend with VectorAI DB.

Create a file index.ts:

import { defineToolPlugin } from "openclaw/plugin-sdk/tool-plugin";
import { Type } from "typebox";
import { VectorAIClient, hasId } from "@actian/vectorai-client";


const DEFAULT_GRPC = "localhost:6574";
const DEFAULT_REST = "http://localhost:6573";
const DEFAULT_OLLAMA = "http://localhost:11434";
const DEFAULT_COLLECTION = "openclaw-memory";
const EMBED_MODEL = "nomic-embed-text";
const DEFAULT_LIMIT = 5;
const EMBED_DIM = 768;


function hashId(input: string): number {
   return Math.abs(
       input.split("").reduce((acc, ch) => (Math.imul(31, acc) + ch.charCodeAt(0)) | 0, 0)
   );
}


async function embed(text: string, ollamaHost: string): Promise<number[]> {
   const res = await fetch(`${ollamaHost}/api/embeddings`, {
       method: "POST",
       headers: { "Content-Type": "application/json" },
       body: JSON.stringify({ model: EMBED_MODEL, prompt: text }),
   });
   if (!res.ok) throw new Error(`Ollama embed failed: ${res.status} ${res.statusText}`);
   return (await res.json()).embedding as number[];
}


export default defineToolPlugin({
   id: "vectorai-memory",
   name: "VectorAI Memory",
   description: "Routes OpenClaw memory tools to Actian VectorAI DB via Ollama embeddings.",


   configSchema: Type.Object({
       grpcAddr: Type.Optional(
           Type.String({ description: "VectorAI gRPC address (default: localhost:6574)" })
       ),
       restUrl: Type.Optional(
           Type.String({ description: "VectorAI REST URL (default: http://localhost:6573)" })
       ),
       ollamaHost: Type.Optional(
           Type.String({ description: "Ollama host for embeddings (default: http://localhost:11434)" })
       ),
       collection: Type.Optional(
           Type.String({ description: "VectorAI collection name (default: openclaw-memory)" })
       ),
   }),


   tools: (tool) => [


       tool({
           name: "vectorai_store",
           label: "Store in VectorAI",
           description: "Embed and store a text chunk in VectorAI DB. Returns the assigned numeric ID.",
           parameters: Type.Object({
               text: Type.String({ description: "The text content to embed and store." }),
               source: Type.String({ description: "The source file or identifier this text came from." }),
               line: Type.Optional(Type.Number({ description: "Line number within the source file." })),
           }),
           async execute({ text, source, line }, config, ctx) {
               ctx?.signal?.throwIfAborted();


               const grpcAddr = config?.grpcAddr ?? DEFAULT_GRPC;
               const restUrl = config?.restUrl ?? DEFAULT_REST;
               const ollamaHost = config?.ollamaHost ?? DEFAULT_OLLAMA;
               const collection = config?.collection ?? DEFAULT_COLLECTION;


               const client = new VectorAIClient(grpcAddr, { restUrl });
               const vector = await embed(text, ollamaHost);
               const id = hashId(`${source}:${line ?? 0}:${text.slice(0, 40)}`);


               await client.points.upsert(collection, [
                   { id, vector, payload: { text, source, line: line ?? 0 } },
               ]);


               return JSON.stringify({ stored: true, id, collection });
           },
       }),


       tool({
           name: "vectorai_search",
           label: "Search VectorAI",
           description: "Semantic search over stored memories. Returns ranked results with scores.",
           parameters: Type.Object({
               query: Type.String({ description: "Natural language query to find semantically similar stored entries." }),
               limit: Type.Optional(
                   Type.Number({
                       description: `Max results to return (default: ${DEFAULT_LIMIT}, max: 20).`,
                       minimum: 1,
                       maximum: 20,
                   })
               ),
           }),
           async execute({ query, limit }, config, ctx) {
               ctx?.signal?.throwIfAborted();


               const grpcAddr = config?.grpcAddr ?? DEFAULT_GRPC;
               const restUrl = config?.restUrl ?? DEFAULT_REST;
               const ollamaHost = config?.ollamaHost ?? DEFAULT_OLLAMA;
               const collection = config?.collection ?? DEFAULT_COLLECTION;


               const client = new VectorAIClient(grpcAddr, { restUrl });
               const vector = await embed(query, ollamaHost);
               const hits = await client.points.search(collection, vector, {
                   limit: limit ?? DEFAULT_LIMIT,
               });


               return JSON.stringify(
                   hits.map((h: any) => ({
                       id: h.id,
                       score: h.score,
                       text: h.payload?.text ?? null,
                       source: h.payload?.source ?? null,
                       line: h.payload?.line ?? null,
                   }))
               );
           },
       }),


       tool({
           name: "vectorai_recall",
           label: "Recall from VectorAI",
           description: "Retrieve a specific memory by its numeric ID.",
           parameters: Type.Object({
               id: Type.Number({ description: "The numeric ID of the memory point to retrieve." }),
           }),
           async execute({ id }, config, ctx) {
               ctx?.signal?.throwIfAborted();


               const grpcAddr = config?.grpcAddr ?? DEFAULT_GRPC;
               const restUrl = config?.restUrl ?? DEFAULT_REST;
               const collection = config?.collection ?? DEFAULT_COLLECTION;


               const client = new VectorAIClient(grpcAddr, { restUrl });
               const hits = await client.points.search(
                   collection,
                   new Array(EMBED_DIM).fill(0),
                   { limit: 1, filter: hasId([id]) }
               );


               const h = hits[0];
               return JSON.stringify(
                   h?.payload
                       ? { id: h.id, text: h.payload.text, source: h.payload.source, line: h.payload.line }
                       : null
               );
           },
       }),


       tool({
           name: "vectorai_forget",
           label: "Forget from VectorAI",
           description: "Permanently delete a stored memory by its numeric ID.",
           parameters: Type.Object({
               id: Type.Number({ description: "The numeric ID of the memory point to delete." }),
           }),
           async execute({ id }, config, ctx) {
               ctx?.signal?.throwIfAborted();


               const grpcAddr = config?.grpcAddr ?? DEFAULT_GRPC;
               const restUrl = config?.restUrl ?? DEFAULT_REST;
               const collection = config?.collection ?? DEFAULT_COLLECTION;


               const client = new VectorAIClient(grpcAddr, { restUrl });
               await client.points.delete(collection, { ids: [id] });


               return JSON.stringify({ deleted: true, id, collection });
           },
       }),


   ],
});

Run the command:

npm run build

This creates the file dist/index.js.

The plugin uses OpenClaw's plugin SDK and exports a standard plugin entry.
The defineToolPlugin() function is the plugin entry point. OpenClaw calls it during startup and exposes the runtime API used to register tools and lifecycle hooks.

The plugin creates a VectorAI DB client and an embedding helper that generates vectors using Ollama's nomic-embed-text model. Every memory operation uses the same embedding pipeline, which keeps retrieval and storage consistent.

Tool implementations

The plugin registers four tools that mirror OpenClaw's memory operations.

vectorai_store embeds a text chunk and stores it together with metadata such as the source file path and line number.

vectorai_search embeds the query, performs a vector search against the openclaw-memory collection, and returns the most relevant memories ranked by semantic similarity.

vectorai_recall retrieves a specific memory by ID.

Each stored record contains metadata. This metadata makes it possible to trace search results back to the original memory file.

vectorai_forget removes a stored memory from the VectorAI DB collection.

At this point, OpenClaw can already store, search, recall, and delete memories through VectorAI DB instead of SQLite/sqlite-vec.

Configure OpenClaw

Create an openclaw.plugin.json file that registers the plugin:

{
   "id": "vectorai-memory",
   "name": "VectorAI memory",
   "version": "1.0.0",
   "description": "Store and retrieve data in Actian VectorAI DB using semantic vector search.",
   "activation": {
       "onStartup": true
   },
   "contracts": {
       "tools": [
           "vectorai_store",
           "vectorai_search",
           "vectorai_recall",
           "vectorai_forget"
       ]
   },
   "configSchema": {
       "type": "object",
       "additionalProperties": false,
       "properties": {
           "grpcAddr": {
               "type": "string"
           },
           "restUrl": {
               "type": "string"
           },
           "ollamaHost": {
               "type": "string"
           },
           "collection": {
               "type": "string"
           }
       }
   }
}

This single configuration change swaps the memory backend from SQLite/sqlite-vec to VectorAI DB. The agent behavior and prompts stay the same but the memory layer changes.

Installing and activating the plugin

At this point, the plugin is complete. The final step is swapping OpenClaw from the default memory-core backend to the new VectorAI DB backend.
So far, your directory structure should look like this:
.

├── create-collection.ts
├── data
├── dist
│ └── index.js
├── docker-compose.yml
├── index.ts
├── openclaw.plugin.json
├── tsconfig.json
├── package-lock.json
├── package.json

Install the plugin from your local directory:

openclaw plugins install .

You see an output shown:

Figure 3: OpenClaw plugin installation

Configure your OpenClaw agent to use the Together AI API Llama-3.3-70B-Instruct-Turbo chat model by default. Replace the placeholder with your API key.

openclaw onboard --non-interactive --accept-risk --mode local --auth-choice together-api-key --together-api-key "$TOGETHER_API_KEY"
openclaw config set agents.defaults.model.primary "together/meta-llama/Llama-3.3-70B-Instruct-Turbo"
openclaw config set plugins.allow '["vectorai-memory", "together"]'

Restart the OpenClaw gateway:

openclaw gateway restart

Test connection

Check the agent your OpenClaw is using by:

openclaw agents list

You should see an output shown below:

" width="512" height="93">
Figure 4: Getting the agent

In this case, OpenClaw uses the default agent main.

To verify the plugin works, store a test memory in VectorAI DB using the plugin:

openclaw agent --agent main --message '/tool vectorai_store {"text":"Rome is the capital of Italy","source":"test","line":1}'

You should see:

" width="512" height="77">
Figure 5: Storing to VectorAI DB

To retrieve it, ask:

openclaw agent --agent main --message '/tool vectorai_search {"query":"what is the capital of Italy?"}'

We get the following results:

" width="512" height="110">
Figure 6: Agent retrieving from VectorAI DB

Why On-Premises Memory Matters for Production Agents

For many teams, cloud-hosted memory services are a practical choice. They simplify deployment and reduce operational overhead. However, they are not always an option.

Production agents often run in environments where data cannot leave the network boundary. Financial institutions, healthcare organizations, government agencies, and industrial environments frequently restrict cloud egress or require local data residency. In these cases, the memory layer must run alongside the application infrastructure.

VectorAI DB allows teams to keep semantic search and durable memory inside their own environment. Memory remains available across agent restarts, context compaction events, and multiple agent instances without depending on an external service.

Wrapping Up

OpenClaw's default SQLite/sqlite-vec memory backend works well for a single agent running on a single machine. As soon as you need shared memory, durable storage across context compaction events, or independent management of the vector store, an external backend becomes a better fit.

In this tutorial, you built a custom OpenClaw plugin that routes memory operations to VectorAI DB. The swap required only a plugin installation, a configuration change, and a gateway restart. The agent behavior stayed the same, but the memory layer became persistent, shareable, and independent of the agent runtime.

Get started with Actian VectorAI DB Community Edition by signing up today. Check the documentation for deployment and usage instructions, and participate in the Discord community for support and discussions.

Running Gemma 2B on Edge Hardware with Actian VectorAI DB

Praise James — Thu, 09 Jul 2026 17:49:35 +0000

Today, running a powerful language model entirely on edge hardware is no longer the hard part. The problem is making it useful once it is there. A developer can deploy Gemma 2B on NVIDIA Jetson Orin Nano or a custom hardware device, run inference locally, and generate responses without sending a single token to the cloud.

The model performs well, supports an 8K token context window, and delivers enough throughput for many production edge applications. The real challenge appears when the agent needs to answer questions from a 50,000-document maintenance corpus that is far larger than anything that fits inside its context window.

A local model without a local vector store faces two choices. Either overload the context window with documents that do not fit or call a remote retrieval service that may not exist in an offline environment. Neither option works for industrial systems, robotics platforms, field service devices, or other regulated industries where connectivity is unreliable or unavailable.

This article builds a complete local inference stack using the Gemma 2B model. Gemma 2B handles text generation through Ollama. Actian VectorAI DB handles semantic retrieval from a large document corpus. The entire stack runs on the device, requires no cloud services during operation, and provides a practical foundation for retrieval-augmented agents deployed at the edge.

What Gemma 2B Brings to Edge Hardware

Most edge AI developers no longer struggle to run a language model locally. The challenge is finding one that delivers useful reasoning and text generation while fitting within the memory and power constraints of edge devices.

Gemma 2B is a lightweight open model designed for local inference. It can run on devices such as the NVIDIA Jetson Orin Nano, Raspberry Pi 5, and industrial gateways. In this setup, the model runs on-device, so data stays local instead of flowing through cloud infrastructure.

Model: Gemma 2B
Parameters: 2 billion
Disk size: ~1.6GB
Target hardware: Raspberry Pi 5, Jetson Orin Nano, edge gateways, embedded Linux devices
RAM required: 8GB+

For many edge workloads, these hardware requirements are modest enough to leave room for additional services on the same device.

Gemma 2B also supports an 8K token context window. While that is sufficient for conversations, instructions, and small document sets, it quickly becomes a limitation when an agent must search across thousands of maintenance manuals, support articles, operational procedures, or historical records. A single industrial knowledge base can contain millions of words, far exceeding what the model can hold in context at once. The model is released under a commercially friendly license, which matters if you need to deploy it on-premises or at the edge.

The problem is that local inference only solves half of the architecture. Once the corpus grows beyond what fits in the context window, the agent needs a retrieval layer that can find the right information before generation begins.

The Retrieval Gap

Getting Gemma 2B running locally solves the generation problem. It does not solve the retrieval problem. An edge agent can only answer questions using information that exists in its prompt or context window. As soon as the knowledge base grows beyond what fits in memory, the agent needs a way to locate the right information before generating a response.

This approach has some failure modes.

Context window overflow

An 8K token context window sounds large until the agent must work with a real-world document collection. A library of regulatory documentation can contain hundreds of thousands or even millions of words.

Without retrieval, developers often attempt to load as many documents as possible into the prompt. That approach quickly reaches the context limit. The model then truncates documents, loses important details, or generates answers from incomplete information. The result is lower accuracy and less reliable responses.

Cloud dependency

Many Retrieval-Augmented Generation (RAG) tutorials solve the context problem by connecting the model to a cloud-hosted vector database. That works in environments where internet connectivity is not a core requirement. It becomes a problem when the application runs in a highly regulated or an air-gapped environment, with no internet access.

When the retrieval layer depends on a cloud service, the agent depends on the network. If the connection fails, the retrieval fails. In many cases, the agent cannot answer questions because the information it needs never reaches the model.

Scale degradation with basic embedding search

Simple embedding search implementations work well with a few hundred documents. Many developers start with an in-memory vector index or a basic search capability bundled with a local inference framework. Performance is acceptable during testing because the dataset is small.

The situation changes when the corpus grows to tens of thousands of document chunks and the application begins serving multiple queries per second. Search latency increases, memory usage rises, and retrieval becomes the bottleneck in the system.

An edge-compliant vector store deployed on the device resolves all three challenges without adding cloud dependency.

The Full Stack

The goal is to keep both inference and retrieval on the same device. Gemma 2B handles generation while VectorAI DB handles retrieval. Together, they form a complete RAG stack that operates without cloud dependency.

In this architecture, the user never interacts with the language model directly. Every query first passes through the retrieval layer, which searches the local document corpus and returns the most relevant information. The application injects those retrieved documents into the prompt before sending it to Gemma 2B. The model then generates a response grounded in the retrieved context rather than relying solely on its training data.

The stack consists of two core components:

Gemma 2B

Gemma 2B serves as the generation layer. It receives the user query along with the retrieved context and produces the final response. Running through Ollama simplifies deployment on edge hardware by providing a lightweight local API for inference.

Responsibilities:

Accept user prompts
Process retrieved context
Generate grounded responses
Run entirely on-device

VectorAI DB

VectorAI DB serves as the retrieval layer. It stores document chunks and their metadata, then performs semantic search against the local corpus. When a user submits a query, the application converts that query into an embedding and sends it to VectorAI DB. The database returns the most semantically relevant document chunks, which become context for the language model.

Responsibilities:

Store document embeddings
Perform a semantic similarity search
Retrieve relevant context
Operate without network connectivity

This architecture keeps every component inside the device boundary. Once the model, embeddings, and database are installed, the system can answer questions without relying on external APIs, cloud-hosted vector databases, or internet connectivity.

Figure 1: Local inference and retrieval stack architecture

Setting Up the Stack

This section builds a complete local retrieval and inference stack on an edge device using Gemma 2B for inference and VectorAI DB for retrieval.

Stage 1: Install Ollama and pull Gemma 2B

Install Ollama on your custom hardware device by following the installation instructions for your operating system.

After installation, pull the Gemma 2B model. Use the latest available tag for Gemma 2B in Ollama. Confirm the exact tag in the Ollama model registry before production use.

ollama pull gemma2:2b

You should see an output:

" width="512" height="76">
Figure 2: Ollama pull for Gemma 2B

Verify the installation by running the test prompt

ollama run gemma2:2b "Explain what a vector database does in one sentence."

You get the result:

Figure 3: Verify Ollama Gemma 2B installation

Stage 2: Install Docker and run VectorAI DB

Install Docker by following the guide and confirm that the Docker service is running. Then, create a docker-compose.yml file with the following configuration:

services:
 vectorai:
   image: actian/vectorai:latest
   container_name: vectorai_db
   ports:
    - "6573:6573" #rest
    - "6574:6574" #grpc
   volumes:
     # vector data persists across restarts
    - ./data:/var/lib/actian-vectorai
   environment:
    - VECTORAI_LOG_LEVEL=info
    - ACTIAN_VECTORAI_ACCEPT_EULA=YES
   restart: unless-stopped

Start the VectorAI DB server by:

docker-compose up -d

Stage 3: Connect with VectorAIClient and create a collection

Install UV and run the command:

uv init .

Install the dependencies by running the command:

uv add actian-vectorai-client requests

Create a file create_collection.py with the following contents:

import requests
from actian_vectorai import VectorAIClient, VectorParams, Distance
# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434/api/embeddings"
OLLAMA_MODEL = "gemma2:2b"
# ──────────────────────────────────────────────────────────────────────────────
def get_embedding_dim() -> int:
   """Probe Ollama once to get the embedding dimension for this model."""
   response = requests.post(
       OLLAMA_URL,
       json={"model": OLLAMA_MODEL, "prompt": "probe"},
       timeout=30,
   )
   response.raise_for_status()
   return len(response.json()["embedding"])
def main():
   print(f"Probing embedding dimension from Ollama ({OLLAMA_MODEL})...")
   dim = get_embedding_dim()
   print(f"  Embedding dimension: {dim}")
   with VectorAIClient(VECTORAI_URL) as client:
       info = client.health_check()
       print(f"\nConnected to {info['title']} v{info['version']}")


       existing = client.collections.list()
       if COLLECTION in existing:
           print(f"\nCollection '{COLLECTION}' already exists — nothing to do.")
           return
       client.collections.create(
           COLLECTION,
           vectors_config=VectorParams(size=dim, distance=Distance.Cosine),
       )
       # 5. Confirm using get_info()
       info = client.collections.get_info(COLLECTION)
       print(f"\nCollection created: '{COLLECTION}'")
       print(f"  Vector size : {dim}")
       print(f"  Distance    : Cosine")
       print(f"  Status      : {info.status}")
       print(f"  Points      : {info.points_count}")
       print(f"\nPayload fields written at ingest time:")
       print(f"  content     — chunk text")
       print(f"  source      — source filename")
       print(f"  chunk_index — position of this chunk within the document")
       print(f"  metadata    — dict with char_start and total_chunks")
if __name__ == "__main__":
   main()

create_collection.py does the following:

Detects embedding size automatically by sending a sample text to Ollama, avoiding hardcoded vector dimensions
Connects to VectorAI DB via gRPC and checks that the database server is running
Checks for an existing docs collection and exits if it already exists, making the script safe to rerun
Creates the docs collection only when it does not exist.

Run this file by:

uv run create_collection.py

You get the following output:

Figure 4: Run create_collection.py

Stage 4: Ingest a document corpus

With the collection created, the next step is to fill it. The ingestion script reads every .txt file in a local docs/directory, splits each file into chunks, generates an embedding for each chunk using Ollama, and writes the result to VectorAI DB. No external API is called at any point in this process.
Create a docs/ directory and add a file named corpus.txt inside it with the following content:


Hydraulic Press Unit 7 — Maintenance Reference


Unit 7 is a 50-ton hydraulic press manufactured by Duratek in 2019.
Serial number: DT-7740-B. Located in Bay 3, Building 2.


Scheduled maintenance is every 90 days. Last service was completed on March 14, 2026 by technician R. Okafor.


Hydraulic fluid: ISO 46 mineral oil. Tank capacity is 12 liters.
Replace fluid every 180 days or if colour turns dark brown.


The main pump runs at 210 bar operating pressure. Maximum rated pressure is 250 bar.
If pressure drops below 180 bar during operation, inspect the pump seals.


Hydraulic coupling torque spec: 42 Nm ± 2 Nm. Use thread-lock grade 243 after torquing.


Known issue: the pressure relief valve on Unit 7 sticks occasionally at cold start.
Workaround is to run the press unloaded for 2 minutes before applying load.
Replacement valve part number: DT-PRV-114.


Emergency stop is the red panel on the left side of the frame.
Do not operate the press without the safety guard in place.

Create a file ingest.py with the following content:

import uuid
from pathlib import Path


import requests
from actian_vectorai import VectorAIClient, PointStruct
# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434/api/embeddings"
OLLAMA_MODEL = "gemma2:2b"
DOCS_DIR     = Path("./docs")
CHUNK_SIZE   = 400    # characters per chunk
# ──────────────────────────────────────────────────────────────────────────────
def embed(text: str) -> list[float]:
   response = requests.post(
       OLLAMA_URL,
       json={"model": OLLAMA_MODEL, "prompt": text},
       timeout=60,
   )
   response.raise_for_status()
   return response.json()["embedding"]
def chunk_text(text: str) -> list[str]:
   return [text[i : i + CHUNK_SIZE] for i in range(0, len(text), CHUNK_SIZE)]
def main():
   files = list(DOCS_DIR.glob("*.txt"))
   if not files:
       print(f"No .txt files found in {DOCS_DIR}/")
       print("Create the folder and drop some .txt files in it, then re-run.")
       return
   print(f"Found {len(files)} file(s) in {DOCS_DIR}/\n")
   with VectorAIClient(VECTORAI_URL) as client:
       # Confirm the collection is there before doing any work
       existing = client.collections.list()
       if COLLECTION not in existing:
           print(f"Collection '{COLLECTION}' not found.")
           print("Run 'python create_collection.py' first.")
           return
       total_chunks = 0
       for path in files:
           text = path.read_text(encoding="utf-8")
           chunks = chunk_text(text)
           points = []
           print(f"Ingesting {path.name} ({len(chunks)} chunks)...")
           for idx, chunk in enumerate(chunks):
               vector = embed(chunk)
               points.append(
                   PointStruct(
                       id=str(uuid.uuid4()),
                       vector=vector,
                       payload={
                           "content":     chunk,
                           "source":      path.name,
                           "chunk_index": idx,
                           "metadata": {
                               "char_start":   idx * CHUNK_SIZE,
                               "total_chunks": len(chunks),
                           },
                       },
                   )
               )
           client.points.upsert(COLLECTION, points)
           total_chunks += len(points)
           print(f"  ✓ {len(points)} chunks written")
       # Final count from the DB
       count = client.points.count(COLLECTION)
       print(f"\nDone. {total_chunks} chunks ingested this run.")
       print(f"Total points in '{COLLECTION}': {count}")
if __name__ == "__main__":
   main()

The ingest.py script is responsible for preparing documents for retrieval. It performs the following steps:

Loads documents from the configured source directory
Splits each document into 400-character chunks to create focused passages that can be retrieved accurately
Generates embeddings for each chunk by sending the text to Ollama's /api/embeddings endpoint
Builds a payload containing the chunk text and associated metadata, such as the document ID, chunk ID, and source information
Stores the vector and payload in the vector database, allowing the chunks to be searched later using semantic similarity

After ingestion is complete, every chunk is represented by both its embedding vector and its metadata. This enables the retrieval system to find the most relevant passages for a query and return the original text along with information about where it came from.

Run the ingest.py by:

uv run ingest.py

This returns the following results:

Figure 5: Run ingest.py

Building the Retrieval Loop

The retrieval loop is the core of the system. It connects three local components into one flow:

The user query input
VectorAI DB for semantic retrieval
Gemma 2B via Ollama for response generation

The full pipeline runs entirely on-device and turns a raw question into a grounded answer using retrieved context.

Create a file query.py with the following content:

import sys
import requests
from actian_vectorai import VectorAIClient


# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434"
OLLAMA_MODEL = "gemma2:2b"
TOP_K        = 3
# ──────────────────────────────────────────────────────────────────────────────




# Step 1: Accepts a user query
def main(query: str):
   print(f"\nQuery: {query}\n")


   # Step 2: Embed the query using the same model used at ingest time.
   # Using the same model is critical — the query vector must exist in the
   # same vector space as the stored chunk vectors for similarity to be meaningful.
   query_vector = embed(query)


   # Step 3: Search VectorAI DB for the top-k most similar chunks
   with VectorAIClient(VECTORAI_URL) as client:
       results = client.points.search(
           COLLECTION,
           vector=query_vector,
           limit=TOP_K,
       )


   if not results:
       print("No results returned. Make sure you have run ingest.py first.")
       return


   # Print retrieved chunks so retrieval is visible, not just implied
   print(f"── Retrieved {len(results)} chunk(s) ───────────────────────────")
   context_parts = []
   for i, result in enumerate(results, 1):
       source  = result.payload.get("source", "unknown")
       chunk   = result.payload.get("chunk_index", "?")
       content = result.payload.get("content", "")
       print(f"[{i}] {source} / chunk {chunk}  (score: {result.score:.3f})")
       print(f"    {content[:120]}...")
       context_parts.append(f"[{source}]\n{content}")
   print("────────────────────────────────────────────────────\n")


   # Step 4: Inject retrieved chunks into the prompt as context.
   # The model is explicitly told to use only the provided context.
   # This keeps the response grounded in the indexed documents and prevents
   # the model from drawing on its general training knowledge.
   context = "\n\n".join(context_parts)
   prompt = (
       "You are a maintenance assistant. "
       "Use only the context below to answer the question. "
       "If the answer is not in the context, say so.\n\n"
       f"Context:\n{context}\n\n"
       f"Question: {query}\n\n"
       "Answer:"
   )


   # Step 5: Call Gemma 2B via the Ollama API and return the response
   print("Generating answer with Gemma 2B...")
   response = requests.post(
       f"{OLLAMA_URL}/api/generate",
       json={"model": OLLAMA_MODEL, "prompt": prompt, "stream": False},
       timeout=120,
   )
   response.raise_for_status()
   answer = response.json()["response"].strip()
   print(f"\nAnswer:\n{answer}\n")




def embed(text: str) -> list[float]:
   response = requests.post(
       f"{OLLAMA_URL}/api/embeddings",
       json={"model": OLLAMA_MODEL, "prompt": text},
       timeout=60,
   )
   response.raise_for_status()
   return response.json()["embedding"]




if __name__ == "__main__":
   if len(sys.argv) < 2:
       print("Usage: python query.py \"your question here\"")
       sys.exit(0)
   main(" ".join(sys.argv[1:]))

This code does the following:

Accepts a user query: The script starts by taking a question from the command line.
Converts the query into an embedding: The embed() function sends the query to Ollama’s embeddings endpoint.
Searches VectorAI DB for relevant context: The embedding is used to query the local vector database.
Prints retrieved chunks: Before generating a response, the script prints the retrieved results.
Builds a grounded prompt for Gemma 2B: All retrieved chunks are merged into a single context block.
Sends the prompt to Gemma 2B via Ollama: The final step sends the structured prompt to the local model.
Returns the final answer: The response is extracted and printed.

Test the end-to-end flow by running the command:

uv run query.py "what is the torque spec for the hydraulic coupling?"

You get the result:

" width="512" height="295">
Figure 6: Building the retrieval loop

From the output, you see that the model is no longer answering from its internal training data alone. Instead, it is explicitly grounded in the retrieved chunks printed from VectorAI DB before generation. Each response is tied to specific documents, which means you can trace every answer back to the source material.

Running Offline

This is the final validation of the entire stack. At this point, both Gemma 2B and VectorAI DB are already installed, the corpus has been indexed, and the retrieval loop is working on a live connection. The next step is to prove that nothing in the runtime path depends on the internet.

Disable your network interface by running the command:

networksetup -setairportpower en0 off  # for MacOs
sudo ip link set eth0 down  # for Linux

Run the command:

uv run query.py "what is the serial number"

We get the following results:

Figure 7: Verify offline execution

Wrapping Up

This system demonstrates that edge AI is no longer limited to isolated model inference. Gemma 2B handles local text generation efficiently on devices like the Jetson Orin Nano, while VectorAI DB provides a persistent semantic memory layer that runs on the same hardware. Together, they form a complete retrieval-augmented generation stack that operates without cloud services during runtime.

After initial setup, the stack continues to function without network access. The model does not require external APIs, and the vector database does not depend on cloud infrastructure. This makes the architecture suitable for industrial environments, robotics systems, and field devices where connectivity is unreliable or restricted.

Best Vector Databases for AI Agents in 2026

Praise James — Thu, 09 Jul 2026 15:51:54 +0000

Evaluating the best vector databases for AI agents in 2026 presents a different problem than it did a few years ago. Most vector databases are adequate in a Retrieval-Augmented Generation (RAG) demo. The decision comes down to fit: where the agent runs, what it needs to do, and how much write load the architecture generates.

Two distinctions matter more than most comparison articles acknowledge. First, RAG retrieval and agent memory are different workloads. RAG systems primarily read from a vector store, while agent memory systems continuously write new observations back into it. Second, deployment constraints often eliminate options before performance becomes relevant. A cloud-managed database may perform well in benchmarks but fail a compliance review or data residency requirement.

This article covers eight databases: Pinecone, Milvus, Qdrant, Weaviate, Chroma, pgvector, LanceDB, and Actian VectorAI DB. It opens with five evaluation criteria, works through the strengths and limitations of each database, maps each to either RAG retrieval or agent memory, and closes with a decision table.

What Is a Vector Database for AI Agents?

A vector database stores embeddings, high-dimensional numerical representations of text, images, audio, and other unstructured data, and retrieves them through semantic similarity rather than exact keyword matching. To perform vector similarity search efficiently, vector databases use Approximate Nearest Neighbor (ANN) algorithms such as Hierarchical Navigable Small World (HNSW) and Inverted File Index (IVF).

Traditional relational databases match records by exact value. An AI agent searching for documents related to "equipment failure in cold conditions" will not retrieve a record titled "low-temperature motor fault" unless those exact words appear. A vector database returns semantically similar results regardless of surface wording. That retrieval quality gap is why agents need a dedicated vector store.

AI agents use vector databases for two distinct purposes. The first is RAG, where the agent retrieves relevant documents before generating a response. The second is agent memory, where the agent continuously reads and writes information about previous interactions, observations, and task outcomes.

A vector database does not replace Postgres or another transactional database. Most production distributed architectures store structured records, user metadata, and transactional data in a relational database, while using an AI-native vector database for semantic search, retrieval, and long-term memory operations.

How to Evaluate a Vector Database for AI Agents in 2026

Query speed under load

The fastest vector database benchmark is often the least useful metric for production agents. Static Queries Per Second (QPS) measurements assume a fixed index with no concurrent writes. Real agents continuously ingest documents, update memories, and create embeddings. Streaming ingestion performance at your target recall threshold is often a better production signal than peak benchmark numbers. The VectorDBBench leaderboard, maintained by Zilliz, updates the benchmark suite over time. Because Zilliz also develops Milvus, treat the leaderboard as directional guidance rather than a neutral ranking.

Write pattern support

RAG retrieval is read-heavy. Agent memory is read-write, so every task completion can add write load. VectorDBBench data shows how large this gap can be: Zilliz Cloud drops from 7,385 QPS static to 1,860 QPS with 1,000 rows per second of ingestion, a 75% reduction. Self-hosted Milvus 16c64g drops from 2,747 to 156 QPS under the same ingestion rate. Pinecone p2.x8 drops from 1,131 to 369 QPS under 500 rows-per-second ingestion. Static QPS and streaming ingestion QPS are not the same ranking. A database can look fast in one and fall behind in the other.

Deployment model

A cloud-managed vector database is not always an option for every team. Regulated industries (healthcare under HIPAA, finance under PCI DSS or SOC 2), government systems, and industrial edge deployments have structured data residency requirements that eliminate cloud-managed options entirely. Use three separate categories: cloud-only, self-hosted, and edge or air-gapped. Those deployment constraints determine which databases are relevant for a given team.

Operational overhead

Infrastructure complexity influences long-term costs as much as software pricing. Running pgvector on an existing Postgres instance introduces little additional operational burden. Operating Milvus at scale often requires Kubernetes expertise and dedicated infrastructure support. Managed services reduce operational work but transfer that cost into subscription pricing.

The 8 Best Vector Databases for AI Agents in 2026

The decision depends on three factors: where the agent runs, what it needs to do (retrieval, memory, or both), and how much operational complexity the team can support. Different modern vector databases optimize for different combinations of those requirements.

1. VectorAI DB

VectorAI DB targets environments where deployment constraints matter as much as retrieval performance. Rather than focusing solely on cloud-native workloads, it is designed to run across cloud, on-premises, edge, and air-gapped environments using the same deployment model.
Deployment: Cloud, self-hosted, on-premises, edge, air-gapped environments
Best for:

Regulated industries with strict data residency requirements
Industrial and manufacturing edge AI deployments
Teams that need a consistent deployment model across cloud and disconnected environments

Limitations:

Closed-source engine
Smaller ecosystem than more established vector databases
SDK support is limited to Python and JavaScript at launch

As a newer entrant, VectorAI DB is still expanding its ecosystem, integrations, and platform capabilities. The product roadmap is developing rapidly, with new features and improvements released regularly.

The primary differentiator is deployment flexibility. The same Docker image can run in cloud environments, on-premises infrastructure, NVIDIA Jetson devices, and fully air-gapped networks without requiring different operational models for each environment.

For teams building AI agents in regulated environments, industrial settings, or locations where cloud connectivity is unavailable or restricted, deployment requirements may eliminate many alternatives before benchmark comparisons begin.

2. pgvector

pgvector extends Postgres with vector search capabilities and remains the default choice for teams already invested in the Postgres ecosystem.
Deployment: Self-hosted Postgres
Best for:

Agent memory workloads are tightly coupled with relational data
Teams operating below 10 million vectors

Limitations:

Large vector workloads increase storage overhead and memory pressure
Scaling requires Postgres expertise rather than dedicated vector infrastructure

The strongest argument for pgvector is architectural simplicity. Instead of introducing another database, teams can keep vector search alongside transactional relational data. This approach works particularly well for agents that need frequent joins between memories, user records, and application state.

An independent benchmark from Steezr found pgvector 0.8.0 HNSW performance comparable to Qdrant 1.13 at a 1M vector scale on AWS c6i.2xlarge hardware. That result suggests some teams can stay on pgvector longer before moving to a dedicated vector database.

3. Qdrant

Qdrant balances retrieval performance, deployment flexibility, and support for memory-heavy workloads.
Deployment: Cloud, self-hosted, Kubernetes
Best for:

Agent memory systems with frequent updates
Hybrid search workloads that combine semantic and keyword retrieval

Limitations:

Vector dimensions remain fixed after collection creation
Payload indexes require deliberate planning and configuration

Qdrant's strength lies in versatility. Teams can begin with self-hosted deployments and later adopt managed offerings without changing databases. Its support for metadata filtering, hybrid search capabilities, and memory-oriented workloads has made it a common choice for production AI systems.

A vendor-funded three-week production evaluation by Particula measured Qdrant at 22ms p95 latency on a 10-million-vector workload, compared with 45ms for Pinecone. Engineers should treat the result as directional rather than definitive, as the comparison was not conducted independently.

For teams that need both operational flexibility and solid performance, Qdrant is a good fit.

4. Pinecone

Pinecone targets teams that prefer managed infrastructure and are willing to pay for operational simplicity.
Deployment: Cloud-managed
Best for:

Organizations with limited DevOps capacity
Teams that prioritize managed operations over infrastructure control

Limitations:

No self-hosting option
Costs can increase significantly under large-scale ingestion and retrieval workloads

Pinecone offloads infrastructure management, scaling, and maintenance to the provider. It fits teams that want managed operations and are comfortable trading off self-hosting control.

5. Weaviate

Weaviate is positioned for agentic retrieval workflows and coding-agent integrations.
Deployment: Cloud and self-hosted
Best for:

Agentic retrieval workflows
Coding agents and multi-step reasoning systems

Limitations:

More operational complexity than managed platforms
Kubernetes expertise is often required for larger deployments

Weaviate differentiates itself through agent-native capabilities rather than raw vector search performance. Its Query Agent reached general availability in 2025, and Agent Skills launched in 2026 with integrations targeting Claude Code, Cursor, GitHub Copilot, VS Code, and Gemini CLI.

These key features position Weaviate as more than a storage layer. It increasingly functions as infrastructure for agent orchestration and retrieval workflows.

Teams building sophisticated agent ecosystems may find Weaviate's agent tooling more compelling than benchmark advantages measured in milliseconds.

6. Milvus

Milvus targets teams that need to search billions of vectors across distributed infrastructure. It is a strong fit when scale matters more than operational simplicity.
Deployment: Self-hosted Kubernetes, managed through Zilliz Cloud.
Best for:

Multi-billion vector workloads
Multimodal agents that combine text, image, audio, and video embeddings

Limitations:

Significant operational complexity
Requires immediate patching of critical vulnerabilities such as CVE-2026-26190 on affected versions

Milvus 2.6 also introduced RaBitQ quantization, which claims a 72% memory reduction and 4× faster queries at approximately 95% recall. These figures come from Milvus and should be treated as vendor-published benchmarks.

7. Chroma

Chroma is a local-first option for quickly getting a prototype agent running. Most developers can move from installation to retrieval in minutes.

Deployment: Local-first, single-node
Best for:

Prototyping
Personal AI assistants
Local development environments

Limitations:

Limited production scalability
Weak concurrency handling at larger vector counts

The challenge appears when prototypes become products. A commonly cited practitioner report described Chroma performing well during development but becoming unusable at roughly two million vectors with 12 concurrent users. In that practitioner report, the team said latency dropped from about 800 ms to 28 ms after migrating to Qdrant under the same workload.

For developers looking to validate an idea before committing to infrastructure, Chroma is a practical starting point. For production systems with sustained concurrency or larger vector counts, it is usually a poor fit.

8. LanceDB

LanceDB focuses on embedded AI applications. Rather than running a separate database service, developers can package retrieval directly into local applications.
Deployment: Embedded database, local-first
Best for:

Desktop AI applications
IDE extensions
Offline assistants
Multimodal agent workloads

Limitations:

No built-in multi-tenant access controls
Shorter production history than established alternatives

One notable adoption example comes from Continue, the open-source coding assistant. Continue selected LanceDB because it offered an embedded TypeScript library with fast on-disk retrieval and SQL-style filtering.

That design makes LanceDB particularly attractive when the agent runs on the user's machine rather than inside a centralized cloud platform.

RAG Retrieval vs. Agent Memory: Which Database Fits Which Use Case?

The distinction matters because most evaluation articles treat RAG and agent memory as the same workload, and they are not.

RAG retrieval is read-heavy. Documents are ingested once and queried repeatedly. Freshness matters, but writes happen relatively infrequently.
Agent memory is read-write at runtime. Agents retrieve previous memories, generate new observations, and immediately write those observations back into storage. Freshness becomes a core requirement.

RAG-optimized databases

These databases excel when retrieval dominates writes:

Pinecone
Milvus
pgvector (especially under 10M vectors)

They perform best when agents search document collections rather than constantly updating memory.

Memory-optimized databases

These databases handle continuous writes more effectively:

Qdrant
Weaviate
LanceDB
Actian VectorAI DB

They fit agents that update memory after every interaction, workflow, or task completion.

Both (with caveats)

pgvector works well below about five million vectors when transactional consistency matters.
Chroma works well during development and prototyping, but is not a long-term production memory layer.

One additional consideration is memory freshness. GitHub's engineering team reported using a 28-day auto-expiry policy and repository-scoped memories in GitHub Copilot. That example shows why retention rules matter: stale memories create more risk than missing memories. Any production memory architecture should define retention and expiration policies before optimizing retrieval performance.

Decision Table

This table summarizes the fastest way to narrow your shortlist by workload, deployment, and write pattern.

Database	Deployment	Best for	Hybrid search	Open source / Free	Skip if
pgvector	Self-hosted (Postgres)	Agent memory with relational data	Partial	Yes	You need sustained billion-scale growth, or can't tolerate Postgres operational overhead
Qdrant	Cloud, Self-hosted	Balanced retrieval and memory workloads	Yes	Yes	You need a fully air-gapped managed experience
Pinecone	Cloud	Zero-ops vector search	Yes	No	Data residency or write-heavy workloads matter
Weaviate	Cloud, Self-hosted	Agent-native retrieval pipelines	Yes	Yes	Your team lacks Kubernetes expertise
Milvus	Self-hosted, Managed	Billion-scale search	Yes	Yes	Your team cannot operate a distributed infrastructure
Chroma	Local	Rapid prototyping	Partial	Yes	You expect production concurrency at scale
LanceDB	Embedded	Local and on-device agents	Partial	Yes	You need multi-tenant enterprise controls
VectorAI DB	Cloud, on-premises, edge, air-gapped	Regulated and disconnected deployments	Yes	No	Your deployment is cloud-native, and air-gap support is irrelevant

Wrapping Up

The best vector database for AI agents in 2026 depends more on workload characteristics than features.

If your team already runs Postgres and expects fewer than 10 million vectors, start with pgvector. If you want managed infrastructure, evaluate Pinecone first and compare it with self-hosted Qdrant before committing to the cost-efficiency model.

If your agent runs in a regulated environment, on industrial hardware, or without guaranteed cloud connectivity, VectorAI DB is the only mainstream 2026 database built specifically for that constraint.

Should You Use RAG or Fine-Tune Your LLM?

Offisong Emmanuel — Tue, 19 May 2026 19:43:02 +0000

The debate over retrieval augmented generation (RAG) vs. fine-tuning appears simple at first glance. RAG pulls in external data at inference time. Fine-tuning modifies model weights during training. In production systems, that distinction is insufficient.

According to the Menlo Ventures 2024 State of Generative AI in the Enterprise report, 51 percent of enterprise AI deployments use RAG in production. Only nine percent rely primarily on fine-tuning. Yet research such as the RAFT study from UC Berkeley shows that hybrid systems combining retrieval and fine-tuning outperform either approach alone across benchmarks.

If hybrid systems can produce better results, why does industry adoption favor only RAG? In this article, we’ll compare RAG, fine-tuning, and a hybrid architecture to understand the trade-offs and where each approach excels.

TL;DR

RAG: Best for frequently changing knowledge and moderate traffic; easy to update without retraining.
Fine-tuning: Best for stable domains and high-volume or low-latency tasks; improves task-specific accuracy and formatting.
Hybrid/RAFT: Combines up-to-date retrieval with optimized model behavior for the highest accuracy.
Key trade-off: Choice depends on query volume, how often knowledge changes, and team expertise.

Why the Standard RAG vs. Fine-Tuning Comparison Fails

RAG is a method where the model dynamically pulls in external data at inference time. Each query retrieves relevant documents or knowledge chunks, which the system appends to the prompt, allowing the model to produce answers grounded in current information.

Fine-tuning is the process of modifying a model’s weights during training using labeled data. Instead of relying on external retrieval, the model internalizes patterns directly, producing consistent outputs without querying external sources.

While these definitions are technically correct, most standard comparisons miss the factors that actually drive decisions in production. In real-world systems, the choice between RAG and fine-tuning depends on variables like scale, query volume, and how often your data changes.

Missing variable 1: Context expansion at scale

In many production RAG systems, every request appends hundreds of tokens. That added context changes how the model allocates attention and prioritizes weights.

Large retrieved contexts compete for attention with the prompt and instructions, which can dilute signal quality. Small retrieval errors or loosely relevant chunks can introduce formatting drift, or shift reasoning in subtle ways. The system’s output becomes tightly coupled to retrieval quality.

Fine-tuning works differently. Instead of injecting large volumes of text at inference time, it embeds patterns and constraints directly into the model during training. The distinction affects how the system behaves under real workloads.

Missing variable 2: Retraining frequency

The common advice says “use RAG if knowledge changes frequently" and “use fine-tuning if behavior is stable.” But how frequently is “frequently”?

If your knowledge base changes daily, retraining pipelines may introduce operational friction. Evaluation cycles, dataset versioning, and deployment validation all add delay.

Data preparation also matters. If your organization lacks structured, versioned, and clean datasets, the hidden cost of preparing training data can exceed compute costs.

The Cost Math of RAG vs. Fine-Tuning

Surface-level comparisons of RAG and fine-tuning often ignore the cost curves that determine long-term viability. In production systems, financial estimations are crucial in architectural decisions. To evaluate RAG vs. fine-tuning realistically, we need to examine three cost layers:

Token cost and context expansion
Retrieval infrastructure cost
Training infrastructure cost

The cost structure of RAG

RAG systems introduce a recurring operational cost because each query retrieves external information and injects it into the model’s prompt. That additional context is billed on every request.

Context expansion
Production RAG systems append around 500 tokens of retrieved context to each query. The provider bills those tokens on every request.

Using pricing similar to GPT-5.2 at 1.750 dollars per million input tokens, the incremental monthly cost becomes:

Cost per query
500 tokens × $1.75/1,000,000 = $0.000875 per query

At a small scale, this cost appears negligible. However, because it applies to every query, the total overhead grows linearly with traffic.

At different traffic levels:

Monthly queries	Context cost
10 million	$8,750
50 million	$43,750
100 million	$87,500

This is context overhead alone. It does not include output tokens or base prompt tokens. At a sustained scale, what appears flexible and inexpensive becomes a significant recurring expense.

Vector database and retrieval cost
Token cost is only one component of RAG costs. RAG also relies on a vector database for semantic search. The system must store, index, and query embeddings efficiently.

Public pricing of Pinecone lists:

Storage at approximately 0.33 dollars per gigabyte per month
Read units at approximately 16 dollars per million
Write units at approximately four dollars per million

For example, consider a system handling 50 million queries per month, where each query performs a single vector search (assuming a 1,024-dimension vector). That would result in 50 million read operations monthly. If the system also writes approximately six million records per month, the combined read and write activity would bring the total estimated monthly cost to around $1,532.

At 200 million queries per month, the total expenses rises to $9,000 per month.

Two RAG systems serving identical traffic can therefore have materially different cost structures depending on how the vector database is designed and optimized.

Infrastructure cost
RAG systems require storage and compute infrastructure to generate embeddings, store and index vectors, execute retrieval queries, and run inference. Each of these stages consumes compute resources, typically provisioned through cloud servers that must scale with traffic.

For real-time or high-throughput applications, additional capacity is required to maintain low latency and system reliability. Replication, autoscaling, monitoring, and failover mechanisms all add operational complexity. These infrastructure layers are essential for production-grade RAG, but they expand the total cost footprint beyond token usage alone.

The cost structure of fine-tuning

Fine-tuning introduces a different economic model from RAG systems. Instead of paying incremental costs on every request for external context, you invest upfront to modify the model’s internal behavior.

That upfront investment can be broken into four primary cost categories: data, training compute, experimentation, and operational maintenance.

Data preparation costs
High-quality labeled data is the foundation of effective fine-tuning. This includes collecting domain-specific examples, cleaning inconsistencies, formatting inputs and outputs correctly, and validating annotation quality.

In many organizations, data preparation consumes 20 to 40 percent of the total fine-tuning budget. Poorly curated data directly degrades model performance, leading to additional retraining cycles and wasted compute.

Training compute costs
OpenAI lists fine-tuning at roughly $25 per million training tokens for GPT-4.1. A run using 20 million tokens would cost about $500 in direct training fees, with larger datasets or multiple runs increasing this total.

For self-hosted training, costs depend on model size and hardware. High-performance GPUs such as A100 clusters can cost thousands of dollars per training epoch. Because fine-tuning is rarely a single-pass process, multiple epochs, evaluations, and retraining cycles are common, which further increases the overall cost.

Experimentation and validation costs
Fine-tuning is an iterative process that requires experimentation with hyperparameters, evaluation against baseline models, and testing across edge cases. These workflows require engineering time, infrastructure, and structured evaluation frameworks. Unlike prompt engineering, fine-tuning introduces a full ML lifecycle, adding ongoing operational overhead.

This creates a non-linear cost curve. Fine-tuning concentrates cost at the beginning, while marginal cost per request remains relatively stable as traffic grows.

Whether that trade-off is advantageous depends on three variables: query volume, knowledge stability, and retraining frequency. Without modeling those explicitly, cost comparisons between RAG and fine-tuning remain incomplete.

When RAG Wins

Despite its scaling trade-offs, RAG remains the dominant production choice for a reason. In certain operating conditions, it is structurally more flexible, faster to iterate, and operationally safer than fine-tuning. RAG is suitable in the following scenarios:

1. When knowledge changes frequently

If your domain knowledge changes weekly or daily, fine-tuning becomes operationally expensive. Dataset updates, retraining, evaluation, and deployment introduce delays that can stretch from hours to weeks depending on governance requirements.

Teams frequently underestimate the operational overhead of keeping a fine-tuned model synchronized with a rapidly evolving knowledge base. In these environments, RAG shifts the problem from model retraining to data indexing.

2. When you have extensive unstructured data but limited labeled data

Many organizations possess terabytes of internal documents but lack high-quality supervised datasets. Building labeled training corpora requires annotation workflows, domain experts, and quality validation pipelines. In practice, this often becomes the most expensive part of fine-tuning projects.

RAG bypasses this constraint by allowing models to operate directly on existing document corpora without constructing large labeled datasets.

3. When governance and data residency requirements are strict

Once sensitive information is embedded in model weights, deletion and auditing become difficult. Removing a specific record from a fine-tuned model often requires retraining or maintaining complex dataset lineage.

RAG architectures avoid this issue by keeping sensitive information in external storage systems where standard governance controls already exist.

4. When query volume is moderate

As shown in the earlier cost analysis, context expansion overhead grows with query volume, reaching approximately $43,750 per month at 50 million queries. At moderate traffic, RAG’s per-request costs are typically lower than the amortized expenses of fine-tuning, including training and ongoing maintenance. This makes RAG an attractive choice for organizations that want high-quality outputs without front-loading infrastructure and compute investments.

Use cases

Large-scale examples illustrate RAG’s effectiveness at this volume. Notion’s Q&A assistant is effectively a large-scale RAG system over workspace data. The difficult engineering problem was not retrieval itself, but enforcing identity and access controls during retrieval. When a user queries the assistant, the system must ensure the model only retrieves documents that the user is permitted to see.

LinkedIn leveraged RAG and knowledge graphs to preserve the structure of their support cases. This system retrieved relevant subgraphs rather than isolated text chunks, improving retrieval accuracy by 77.6% and reducing median issue resolution time by 28.6%.

For systems at this scale, RAG combines cost efficiency with flexibility, allowing teams to update knowledge sources rapidly without retraining models, while still delivering high-quality results.

When Fine-Tuning Wins

Fine-tuning becomes structurally advantageous under different conditions. These conditions typically involve scale, stability, and behavioral precision.

1. When query volume exceeds 100 million per month

At very high traffic levels (100M+ queries per month), RAG’s per-request context overhead becomes significant. Each query adds hundreds of retrieved tokens that the model processes, causing costs to scale linearly with traffic. Large context windows can also increase latency, reduce throughput, and complicate infrastructure reliability.

If domain knowledge is relatively stable, fine-tuning can become more efficient. By embedding knowledge directly into the model, organizations avoid repeated retrieval and token costs, leading to more predictable per-query expenses, better consistency, and simpler operations at scale.

2. When output structure is critical

Fine-tuned models often excel in tasks that require strict adherence to structure or formal constraints. For example, Cosine, which is an AI software engineering assistant that’s able to autonomously resolve bugs and build features, was able to achieve a SOTA score of 43.8% on the SWE-bench⁠ verified benchmark.

Similarly, Distyl secured the top position on the BIRD-SQL benchmark, widely regarded as the premier evaluation for text-to-SQL performance. Its fine-tuned GPT-4o model reached an execution accuracy of 71.83% on the leaderboard.

In applications where errors propagate downstream, into financial calculations, automated APIs, or compliance documents, behavioral consistency is mandatory. In these contexts, fine-tuning provides the reliability needed to minimize risk and maintain trust in automated outputs.

3. When latency requirements are strict

RAG adds multiple steps to the inference pipeline that increase response time. Each query must go through embedding generation, vector search, and context injection before reaching the model.

Fine-tuned models skip retrieval entirely. All necessary knowledge and reasoning patterns are internalized, allowing the model to generate outputs immediately. In applications where sub-100ms responses are required, such as live recommendation engines or high-frequency trading systems, removing the retrieval pipeline eliminates a major bottleneck.

4. When deep domain reasoning matters more than freshness

A domain-specific agriculture benchmark study found that fine-tuning improved model accuracy from 75% to 81%, while hybrid systems (fine-tuning + retrieval) reached 86%. Because the dataset focused on specialized agricultural knowledge and reasoning tasks, the improvement primarily reflects stronger domain reasoning, not simply better access to external information.

In domains such as legal analysis or medical decision support, reasoning patterns can be complex. Fine-tuning enables models to internalize domain expertise rather than rely solely on retrieved context.

The Hybrid Approach

While RAG and fine-tuning each have clear advantages, research shows that combining them effectively can produce superior results, but only when done correctly. The RAFT (Retrieval Augmented Fine-Tuning) approach, developed by UC Berkeley, Microsoft, and Meta Research, demonstrates how to do this in practice.

RAFT trains a model to operate in an “open-book” setting. It learns to process retrieved context, identify relevant passages, ignore distractors, and cite evidence accurately. Without this explicit training, simply layering RAG on top of a fine-tuned model often fails. For instance, a model fine-tuned on medical reasoning may retrieve irrelevant journal articles if it hasn’t learned to filter and prioritize context, resulting in hallucinations or incorrect recommendations.

RAFT addresses this with a structured 80/20 training split. 80% of training examples include oracle documents that the model should use, and 20% do not, forcing the model to learn when to trust retrieved data and when to rely on internalized knowledge. This operational detail is crucial for engineers evaluating whether their team can implement a hybrid approach successfully. It is not enough to just combine RAG and fine-tuning. The model must be trained to reason over the retrieved context.

A common and practical pattern is “fine-tune for format, RAG for knowledge.” Fine-tuning shapes the model’s internal behavior, enforcing domain-specific reasoning, output structure, and style. RAG provides dynamic access to external information that changes frequently or is too large to store in the model weights. In healthcare, for example, fine-tuning ensures the model understands medical terminology, follows proper diagnostic reasoning, and formats outputs according to clinical documentation standards. RAG supplements this by retrieving the latest research, newly published treatment guidelines, or patient-specific records, keeping recommendations current without retraining the entire model.

Similarly, Harvey AI fine-tuned on 10 billion case law tokens, but still leverages RAG to handle current cases and updates. This pattern is widely used in other domains too. Legal systems fine-tune for statutory reasoning and citation style, then layer RAG to retrieve the most current case law; finance models fine-tune for portfolio analysis rules, then layer RAG for market updates and regulatory changes. It’s a way to balance the stability of learned behavior with the adaptability of retrieval.

A Quantified Decision Framework for RAG vs. Fine-Tuning

The question is no longer “Which approach is better?” It is “Under what conditions does each approach make economic and operational sense?”

Instead of defaulting to architectural preference, evaluate three measurable variables:

Knowledge change frequency
Monthly query volume
Infrastructure capability and governance constraints

When those variables are quantified, the decision becomes far clearer.

Step 1: Measure knowledge volatility

Knowledge change frequency is often the fastest way to eliminate one option. If your domain knowledge changes weekly or daily, RAG is structurally favored. Updating an index is far simpler than retraining a fine-tuned model. The separation between model weights and external data enables real time data retrieval without redeployment cycles.

If knowledge remains stable for months at a time, fine-tuning becomes economically viable. Retraining frequency drops, and training cost can be amortized over longer intervals. In these environments, embedding domain specific knowledge directly into model parameters may reduce long-term inference overhead.

As a practical threshold:

Knowledge changes more than monthly → prioritize RAG
Knowledge stable for multiple months → evaluate fine-tuning

Step 2: Calculate context expansion cost**

The next variable is query volume. Large-scale RAG systems append hundreds of tokens to every query, and this context overhead scales linearly with traffic.

Quantitative triggers

Monthly queries	Guidance
<10M	RAG is cheaper
10–50M	Evaluate fine-tuning vs. RAG
50–100M	Fine-tuning or hybrid
>100M	Fine-tuning or hybrid

Step 3: Assess infrastructure maturity

Even if economics favor one approach, infrastructure capability may dictate feasibility.

RAG requires:

Strong data engineering
Reliable data pipelines
Efficient vector database architecture
Observability and monitoring

Fine-tuning requires:

High quality labeled data
Machine learning expertise
Compute resource allocation
Evaluation discipline

When teams ignore their actual capabilities, architecture decisions collapse under scale. Many production failures blamed on “model quality” are just traits of immature infrastructure.

Decision matrix

The following matrix translates the analysis into practical guidance.

Scenario	Monthly queries	Knowledge update frequency	Recommendation	Rationale
Domain knowledge updates weekly, moderate traffic	10–50M	Weekly/Daily	RAG	Immediate indexing and low recurring cost
High-scale traffic, knowledge stable	50–100M+	<1 update/month	Fine-tuning	Avoids recurring context injection, reduces latency
Structured output or code generation required	Any	Any	Fine-tuning	Embeds domain-specific rules and formatting internally
Specialized reasoning + frequent updates	10–50M	Weekly/Daily	Hybrid	Combines internalized reasoning with dynamic knowledge
Multi-domain systems with diverse knowledge update cycles	10–100M	Mixed	Hybrid	Fine-tuning stabilizes core domains, RAG handles rapidly changing sources

Using this matrix, it becomes easier to make the decision whether to utilize RAG, fine-tune your LLMs, or use the hybrid approach.

Final Thoughts

The debate between RAG and fine-tuning is often framed as a binary choice, but the more useful question is “If hybrid systems demonstrably outperform either approach alone, why does industry adoption still overwhelmingly favor RAG?”

Hybrid requires both ML and data engineering capabilities simultaneously, a combination few organizations have. RAG remains the practical default, offering agility and transparency with less upfront complexity.

The key takeaway is to choose the architecture that matches your knowledge volatility, query scale, and team capability. For teams exploring enterprise-scale retrieval systems, platforms like Actian VectorAI DB provide purpose-built vector database capabilities designed for performance and scalability.

Join the Discord community and learn how Actian fits to your AI strategy.

What 37signals’ Cloud Repatriation Taught Us About AI Infrastructure

Offisong Emmanuel — Tue, 19 May 2026 19:42:51 +0000

In 2023, 37signals announced that it had completely left the public cloud and followed up by publicly documenting its cloud repatriation process, providing one of the clearest real-world examples of on-premises economics at scale. By reversing its cloud migration and shifting workloads to private cloud infrastructure, the company drastically reduced its annual cloud infrastructure spend by almost $2 million.

The transparency of the numbers made the case compelling. In 2022, 37signals spent $3,201,564 on cloud services, which is about $266,797 per month. These detailed cost breakdowns, along with published hardware investment and payback timelines, provided a rare look into the financial mechanics of large-scale cloud repatriation.

For commodity SaaS workloads, the math was clear. But the same logic raises an important question for the next generation of compute-heavy systems: “Does the economic argument extend to AI infrastructure as well?” In this article, we examine whether the same economic logic holds for AI infrastructure.

TL;DR

37signals spent ~$3.2M/year on AWS in 2022.
After repatriating workloads to their own infrastructure, cloud spend dropped to ~$1.3M by 2024.
The company invested roughly $700K–$800K in servers and paid them off in under 18 months.
The entire infrastructure is still run by the same 10-person team. No additional operational overhead.
The key takeaway is that at a sustained scale, owning infrastructure can be dramatically cheaper than renting it.

The 37signals Playbook: What Hanson Actually Documented

In 2022, 37signals spent $3.2 million annually on AWS. After leaving the cloud in 2023, their annual costs had dropped to approximately $1.3 million by 2024, a reduction of almost $2 million per year.

The transition required a hardware investment of roughly $600,000 in Dell servers. The company fully recouped the investment in under 18 months, achieving complete payback in the second half of 2023 as their AWS reserved instance contracts expired. From that point forward, the savings flowed directly to operating margin rather than offsetting capital expense.

37signals projected $1.5 million in hardware costs and roughly $200,000 per year in operating expenses. This shift replaces a recurring $1.3 million annual cloud storage bill with a one-time capital outlay plus a fraction of the ongoing operating cost. Over five years, 37signals revised the total savings projections upward from $7 million to more than $10 million.

37signals cloud exit financials by year

To illustrate the financial impact of 37signals’ cloud exit over time, the table below breaks down annual cloud spending, on-premises hardware investments, and operating costs, highlighting the resulting net savings and key operational notes.

Year	Cloud spend	Hardware investment	Operating costs	Notes
2022 Baseline	~$3.2M	$0	Included in cloud spend	Full cloud dependency
2023 Migration	~$2M	~$700–800K	Moderate	Hardware fully recouped in under 18 months
2024+ Post-repatriation	~$1.3M	~$1.5M (storage)	~$200K/year	~$1.9M annual savings
2025+	Minimal AWS dependency	~$1.5M (Pure Storage, 18PB)	~$200K/year	$10M+ projected 5-year savings

Notably, the migration did not require the team to expand operations. A 10-person infrastructure team handled the entire repatriation without adding new staff. Addressing a common concern about operational overhead, 37signals co-founder David Heinemeier Hansso n noted:

“We've been out for just over a year now, and the team managing everything is still the same. There were no hidden dragons of additional workload associated with the exit that required us to balloon the team, as some spectators speculated when we announced it. All the answers in our Big Cloud Exit FAQ continue to hold.”

This directly challenges the common assumption that moving away from public cloud environments inevitably requires a significantly larger infrastructure team.

Execution followed a “criticality ladder” strategy where the team migrated lower-risk services first and more critical ones later. The team moved the HEY email system in stages, starting with caching, then database, and finally, job services. To minimize risk, they colocated infrastructure approximately one millisecond from the AWS region to preserve rollback capability during the cloud repatriation process. After stabilizing the system, they replaced managed services with substantial recurring cost, including RDS and managed Elasticsearch which exceeded $500,000 together annually.

What makes 37signals' case study consequential is the publicly documented cost efficiency. For organizations questioning long-term cloud adoption assumptions particularly with regard to storage costs and managed services, the 37signals documentation provides a rare baseline for comparison.

Why AI Infrastructure Economics Are Even More Extreme

The lessons from 37signals’ cloud repatriation take on a sharper edge when applied to AI infrastructure. Higher GPU costs, predictable inference workloads, massive embedding storage, and stricter data regulations create financial and operational pressures that amplify the advantages of on-premises or hybrid cloud solutions that allow you to move workloads where they make the most sense. Below, we break down the key drivers.

AI infrastructure cost comparison

To evaluate the cost implications of different AI infrastructure approaches, the table below compares upfront setup costs, monthly operating expenses at varying workloads, and expected break-even timelines for cloud, on-premises, and hybrid configurations.

Setup	Setup cost	Monthly cost	Break-even
Cloud GPU rental (AWS/ Azure)	$0	$2,900–3,500 (8h/day × $4–8/hour × 15 days)	N/A
Cloud inference APIs (Lambda Labs)	$0	$1,800–2,500 (8h/day × $3.67/hour × 15 days)	N/A
Self-hosted GPU (8×H100 server)	$200K–400K	$1,500–2,000 (power + maintenance)	<12 months
Hybrid (Cloud training + On-Prem)	$200K–400K	Training only, inference minimal	<12 months

Note: For cloud GPU rental, we estimate monthly cost assuming eight hours/day per GPU. The cost scales linearly with utilization; it is not directly per-query.

1. GPU cloud markups are high

AI workloads depend heavily on GPUs, and cloud providers charge far steeper premiums for GPU capacity than for typical CPU compute. On-demand AWS P5 instances with H100 GPUs cost roughly $4–8 per GPU-hour, while comparable Azure H100 instances are about $3.67 per hour. By contrast, spot markets and alternative providers such as Lambda Labs offer similar GPU capacity for $1–2 per hour, or $1.85–2.49 per hour with reserved commitments.

The result is a 4–8× markup for on-demand hyperscaler GPU capacity relative to the spot or specialized GPU cloud market. In other words, the premium cloud providers charge for high-end AI compute is significantly larger than typical CPU cloud markups. For organizations running sustained inference workloads, this pricing gap quickly becomes the dominant cost driver in AI infrastructure.

2. Predictable inference makes GPU ownership economical

High GPU pricing becomes especially significant because AI inference workloads are unusually predictable. Purchasing H100 GPUs outright can be cost-efficient. A single GPU costs roughly $25K–40K, while a complete 8×H100 server ranges from $200K–400K. Lenovo’s analysis shows that six or more hours of sustained daily usage reaches payback against AWS within the first year.

The reason this break-even arrives so quickly is that AI inference workloads are unusually predictable. Unlike SaaS traffic which fluctuates throughout the day, production AI systems such as recommendation engines tend to process steady volumes of requests.

Predictability changes the economics. When infrastructure runs at consistent utilization, owned hardware can be amortized efficiently across the workload. Paying cloud premiums for burst capacity that teams rarely use becomes unnecessary.

For organizations running inference continuously, the hardware investment is often recouped in under 12 months. From that point forward, the savings resemble the same pattern documented by 37signals. Fixed infrastructure replacing an ongoing rental bill.

3. Embedding storage requirements are massive

Even if GPU compute were optimized, AI systems introduce another rapidly growing cost layer: embedding storage. Vector databases store high-dimensional embeddings used for search, retrieval, and recommendation. As datasets scale into millions or billions of records, storage requirements expand quickly.

For instance, 10 million vectors at 1,536 dimensions require at least 58GB of raw storage, often 200–300GB with indexes and metadata. Cloud storage services like Pinecone charge $0.33/GB/month, meaning 500GB could cost $165/month before any queries. Self-hosted solutions like PostgreSQL with pgvector dramatically reduce cloud spending while keeping sensitive data under direct control. Over time, these storage requirements compound infrastructure costs alongside GPU compute, further reinforcing the economic advantages of self-hosted or hybrid architectures.

4. Data sovereignty and compliance favor on-premises deployment

Data residency regulations and general compliance are priorities in the AI space with the industry becoming increasingly regulated. Notably, the EU AI Act introduced strict regulations for AI systems, with prohibitions on certain AI use cases which took effect in February 2025. On-premises deployment simplifies compliance.

For financial organizations navigating complex regulatory environments, solutions like Actian’s Data Intelligence platform helps enforce data governance and streamline compliance workflows.

The Cloud Infrastructure Case Studies 37signals Validated

As much as the financial transparency of 37signals’ cloud exit was radical, their repatriation was not an isolated occurrence. It was part of a growing trend by many organizations trying to regain cost control and optimize their cloud infrastructure. Many high-profile case studies illustrate the scale and economics of moving workloads back from public clouds to owned or hybrid infrastructure.

Dropbox

Dropbox pioneered enterprise cloud repatriation as early as 2015, completing the migration between 2016 and 2018. The company moved roughly 90% of customer data, reportedly over 500 petabytes, off AWS to three owned colocation facilities. The infrastructure investment totaled $53 million, yet Dropbox reported $74.6 million in operational savings over two years per its 2018 S‑1 filing. A small portion of workloads, primarily European customers and specialized services, remain in AWS. Internally, the initiative was known as “Magic Pocket,” and it exemplifies how a well-executed hybrid cloud approach can deliver substantial savings while aligning with long-term business objectives.

Ahrefs

Ahrefs, the SEO tools company, relied on a Singapore colocation setup with 850 servers. Their reported savings from avoiding public cloud were approximately $400 million over 2.5 years. Actual infrastructure cost: $39.5 million for 850 servers (~$1,500/server/month), versus an estimated $447.7 million if hosted entirely on AWS (~$17,557/server/month equivalent). As Ahrefs put it: “We wouldn’t be profitable, or even exist, if our products were 100% on AWS.” While critics argue that Ahrefs inflated AWS estimates, the directional savings were undeniable, illustrating that cloud repatriation challenges can be surmounted at scale with careful planning.

GEICO

GEICO spent a decade migrating to multiple cloud providers only for its costs to climb and exceed projections by 2.5×, reaching $300 million by 2022 across eight providers. In response, GEICO began moving workloads to a private cloud using OpenStack and Kubernetes, targeting over 50% repatriation by 2029. Early results show 50% reductions in compute and 60% reduction per gigabyte of storage costs compared with public cloud services, demonstrating how a hybrid cloud architecture can deliver efficiency, compliance, and alignment with long-term business objectives.

Akamai

Akamai was on the path to spending over $100 million on third party cloud services before migrating compute workloads to its own global edge network of 350,000+ servers. The migration delivered savings of roughly $100 million per year, a testament to the economics of repatriation when existing infrastructure and scale align.

What these cases share is the same economic pattern documented by 37signals. Predictable, high-volume workloads eventually become cheaper to run on owned infrastructure than on hyperscaler clouds.

These examples reflect a broader shift occurring across enterprise infrastructure strategies. Barclays’ Chief Information Officers (CIO) surveys show cloud repatriation trending upward in recent years, with the sentiment peaking in the second half of 2024 with 86% of CIOs planning repatriation.

However, this statistic does not mean that companies are abandoning public cloud environments completely. According to IDC, only 8–9% of companies favor full repatriation with most preferring a hybrid approach that combines public and private clouds. Hybrid cloud infrastructure allows organizations to optimize workload placement by strategically allocating sensitive data and mission-critical applications on-premises while leveraging public cloud services for less critical workloads. As such, it has become increasingly important for teams exploring similar transitions to understand the nuances of hybrid deployments and their associated risks.

Cloud Repatriation Statistics

Cloud repatriation is accelerating at the same time as public cloud spending keeps climbing. IDC projects global public cloud spend will reach $1.6 trillion in 2028, doubling from their 2024 prediction. Yet as mentioned earlier, 86% of CIOs are planning some form of repatriation according to Barclays. Both trends can be true because this is not a cloud exodus so much as a rebalancing. Enterprises are leaning towards a hybrid cloud model.

AI is likely to accelerate that shift. AI workloads account for less than 10% of total cloud compute today but Gartner projects that this figure will approach 50% by 2029. Hyperscalers are responding with enormous capital investment. There is an estimated $600 billion in infrastructure spend in 2026, roughly three-quarters of it tied to AI. The assumption is clear: Enterprises will rent that GPU capacity. But the 37signals math suggests that once AI workloads move from experimentation to steady production, ownership economics begin to dominate.

Cost pressure is already driving behavior. Flexera reports that 27% of cloud resources are wasted or underutilized, and 21% of workloads have already been repatriated. The primary reason cited is cost exceeding projections, followed by performance concerns. With GPUs, the margin for inefficiency is thinner. There are fewer optimization levers, higher hourly rates, and faster budget burn.

Regulation adds another layer. The EU AI Act, DORA for financial services, China’s PIPL, and India’s DPDP are tightening data governance requirements. Mimecast reports that 87% of organizations now factor data sovereignty into vendor decisions. For AI systems, sovereignty extends beyond data location to model provenance, audit trails, and compliance documentation. On-premises deployment does not eliminate regulatory complexity, but it centralizes control and for many enterprises, that simplicity is becoming strategically attractive.

The Counter-Arguments and When Cloud Providers Win

Not all observers agree that cloud repatriation is the best path for every organization. Public cloud environments still deliver value in certain circumstances. But arguments often do not hold strong in the case of AI workloads.

When cloud wins vs. when on-premises wins

Component	Cloud advantage	On-prem advantage
Workload predictability	Handles spiky or unpredictable workloads	Predictable workloads cheaper to self-host
Team expertise	Requires minimal in-house infrastructure skill	Strong IT teams can optimize and reduce vendor reliance
Scale and growth	Rapid scaling and global expansion	Predictable growth enables cost-efficient hardware
Regulatory requirements	Managed compliance, geo-redundancy	Direct control simplifies regulatory alignment
Cost and margins	Pay-as-you-go reduces upfront spend	Long-term savings from owned infrastructure
Service quality	Cloud SLAs ensure availability and performance	Dedicated resources guarantee predictable uptime

Cloud “wrong usage” argument

Jeremy Daly, a serverless advocate, argues that “37signals was using the cloud wrong.” By treating cloud environments as virtual colocation, running VMs and Kubernetes, they were paying cloud premiums without capturing the value of serverless, managed services, and instant scaling. As Daly notes, “In the cloud, we should be renting services, not servers.”

For SaaS workloads with highly variable or spiky traffic, this argument is compelling. Serverless infrastructure allows organizations to scale instantly and pay only for the compute they actually use.

However, AI inference workloads often behave very differently. Production inference systems, such as recommendation models, copilots, and document processing pipelines, tend to run at steady, sustained utilization rather than unpredictable bursts. In these cases, the economic advantage of elastic cloud scaling diminishes. The premium paid for burst capacity still exists, but the workload itself rarely needs that burst capacity.

Daly’s argument therefore holds for variable SaaS workloads, where elasticity is critical. For sustained AI inference workloads running at high utilization, paying a premium for burst capacity that is rarely used can make dedicated infrastructure or hybrid deployments more cost-efficient.

Full cost critique

Some critics also question the financial assumptions behind 37signals’ approach. They point out that hardware and software normally account for only about 20% of IT costs, with the remainder covering electricity, cooling, physical security, racking, Uninterruptible Power Supply (UPS), and opportunity costs. David Heinemeier Hanson’s analysis did not include all of these overheads because 37signals used colocation facilities rather than fully owned data centers. Even so, considering 37signals’ figures, it is reasonable to conclude that renting colocation space can still be far cheaper than relying on cloud services.

Competence vs. growth framework

Forrest Brazeal’s IT competence versus growth aspirations framework provides additional nuance. He places 37signals in the High Competence/Low Growth quadrant, ideal for self-hosting. “Not every company has the competence (high) or growth aspirations (low) of 37signals,” he observes. Startups with uncertain or spiky workloads benefit from cloud flexibility, but AI companies running production inference at scale often combine high operational competence with steady growth. Such profiles (steady growth & high competence) are well suited to repatriation.

Applying the Playbook to AI Infrastructure

If 37signals provided the economic blueprint, AI infrastructure makes the economics more concrete. The decision is no longer abstract. It becomes a structured assessment grounded in workload behavior, utilization, and regulatory exposure.

A practical four-question framework helps translate the 37signals logic into AI terms:

Is your inference workload predictable and sustained?

Unlike SaaS traffic spikes, most production AI systems such as recommendation engines, RAG pipelines, or fraud detection models process steady volumes with gradual growth.
Are projected GPU utilization rates above 60–70%?

At this threshold, owned hardware amortization typically undercuts public cloud GPU pricing within the first year.
Are you processing more than 10–50 million queries per month?

At this scale, per-token and per-query pricing from cloud APIs compound rapidly.
Do you face data sovereignty or strict compliance requirements?

For financial services, healthcare, or government workloads, regulatory mandates can tilt the decision toward controlled environments.

If the answer is “yes” to three or four of these, the repatriation economics tend to favor on-premises deployment for production inference.

Decision matrix

Workload stage	Recommended environment	Rationale
Model training	Public cloud	Compute-intensive; cloud GPUs handle burst workloads cost-effectively
Experimentation and prototyping	Public cloud	Flexible, fast provisioning for early-stage iteration
Production inference	On-premises / Hybrid	Steady workloads; owned hardware cheaper at 60–70%+ GPU utilization
Vector storage (embeddings)	On-premises	Reduces recurring managed-service costs and ensures data control

The hybrid AI pattern

In practice, most AI organizations adopt a hybrid model rather than an all-or-nothing shift. Training remains in the cloud. Inference moves closer to owned infrastructure.

Lenovo documented that training Llama 3.1 at hyperscale (39.3 million GPU hours) in the cloud would exceed $483 million. That type of elastic, short-term scale is exactly where public cloud excels. Inference is different. Once a model is trained, serving it for three to five years becomes steady, predictable work. That is where amortized hardware economics has the upper hand.

This split architecture also simplifies data migration risk. Instead of relocating entire AI pipelines at once, organizations can migrate production inference workloads gradually while leaving experimentation and early-stage training in cloud environments. A controlled, phased migration process reduces operational disruption while ensuring seamless integration between cloud-based training and on-premises serving layers.

Self-hosted inference economics

The economics of self-hosted inference depend heavily on utilization and token volume. According to enterprise deployment benchmarks, a 7B-parameter model running on an H100 GPU at roughly 70% utilization costs about $10,000 per year in spot nodes or hardware amortization. Power costs about $300 annually, bringing the total costs to about $10,300.

Public LLM APIs, by contrast, typically charge per million tokens, with enterprise pricing in 2025 ranging from $0.25–$15 per million input tokens and $1.25–$75 per million output tokens depending on model tier and provider.

At low usage levels, APIs remain the more economical option because infrastructure sits idle. However, the economics change as workloads scale. Industry analyses suggest that self-hosted deployment begins to break even at roughly two million tokens per day, after which the fixed cost of owned infrastructure is amortized across a large inference volume.

At high volumes, self-hosted inference can reduce costs by up to 78%. Artefact’s analysis) found break-even around 8,000 conversations per day. Below that threshold, managed cloud APIs remain more economical. Above it, ownership compounds savings. The pattern mirrors 37signals: predictable workload plus high utilization equals rapid payback.

Vector databases

Instacart documented migrating from Elasticsearch plus FAISS to PostgreSQL with pgvector, achieving 80% cost savings and a 10× reduction in write amplification. Timescale’s pgvectorscale benchmarks show approximately 75% lower costs than managed vector services like Pinecone at comparable performance.

For RAG systems handling millions of queries monthly, self-hosted vector infrastructure produces savings that resemble the 37signals S3 case: large recurring storage bills replaced by amortized hardware and open-source tooling.

Data sovereignty as a structural driver

Grandview research reports that the sovereign cloud market was worth 648.87 billion USD in 2025 and is projected to reach USD 648.87 billion by 2033. Also, according to Gartner, around 60% of financial firms outside the United States are expected to adopt sovereign or on-premises deployments by 2028.

Frameworks such as the EU AI Act, China’s PIPL, and India’s DPDP mandate data localization and traceability. For organizations processing sensitive training datasets or proprietary inference logs, on-premises deployment inherently satisfies residency requirements because data never leaves jurisdictional boundaries.

The Bottom Line

37signals showed that cloud repatriation teams can measure, model, and defend decisions with hard numbers. With AI infrastructure, the economics can be even more pronounced. If cloud repatriation saved roughly $10 million for Basecamp, an equivalent AI company running production inference at comparable scale could save multiples of that amount, given the much higher cost of GPU compute and embedding infrastructure.

For organizations choosing to run AI workloads in controlled environments, platforms like Actian VectorAI DB provide a purpose-built vector database designed for high-volume vector search and AI inference workloads. It can be deployed on-premises or in the cloud, allowing organizations to place vector infrastructure where it best fits their operational and economic requirements.

Join the community and learn more about Actian.

How to Build a HIPAA Compliant AI Ecosystem Without the Cloud

Offisong Emmanuel — Tue, 19 May 2026 19:42:41 +0000

Healthcare cannot rely on cloud RAG because patient data leaves your network and your system logs, stores, and exposes it outside your control. You sign a Business Associate Agreement (BAA), connect your pipeline to a managed vector database, and assume compliance is complete. That assumption is wrong. The BAA covers the provider’s infrastructure. It does not cover what your application sends, logs, or exposes during retrieval and generation.

You remain responsible for every path your system sends Protected Health Information (PHI) through. A clinician query can leak sensitive data through logs. A system prompt can include patient context that your system stores outside your boundary. Weak access control allows retrieval results to expose records across departments. These risks exist in your application layer, not in the cloud provider’s scope.

American regulators now target this gap. In 2026, they flagged attack patterns like membership inference, where an adversary probes an AI system to confirm whether a patient’s data exists in the index. Cloud-hosted pipelines increase this risk because queries and embeddings move across external infrastructure. Audit requirements tighten further when logs live on third-party systems.

In this tutorial, you will build a clinical knowledge assistant that runs entirely on hospital infrastructure. It performs semantic search over clinical data, enforces role-based access at query time, and generates answers with clear citations. Every query stays inside your network, every access is logged locally, and no external API calls are required.

Why BAA Is Not Enough

A BAA protects the cloud provider’s infrastructure, not how your system handles PHI during queries, retrieval, and generation. You remain responsible for every place PHI appears, moves, or gets stored inside your pipeline. There are multiple failure modes that make your system non-compliant, even when you sign a BAA.

Shared responsibility gap

The BAA stops at the infrastructure boundary. Your system controls what enters a prompt, what gets logged, and what leaves your network. If a clinician query includes PHI and your application logs it to an external service, you are responsible. If your retrieval step returns records across departments without strict filters, you have created an internal data breach. These failures happen in your code, not in the cloud provider’s scope.

For example, a physician searches “Show me similar cases to John Doe with early stage lung cancer.” Your application logs the full query to a cloud logging service for debugging. That log now contains PHI outside your network. The cloud provider did not leak it. Your application sent it.

Audit log ownership

HIPAA requires a complete audit trail for every access to PHI. When your vector database runs on third-party infrastructure, your system stores query logs and retrieval traces outside your control. You cannot guarantee completeness, retention, or isolation. Your security team cannot verify access patterns without relying on another provider’s system. That breaks your ability to enforce and prove compliance.

For example, your compliance team asks for a report of all oncology patient records in the past 30 days. Your vector database provider stores query logs on their platform with limited retention. Some logs are missing and others lack user-level metadata. You cannot produce a complete audit trail.

Membership inference exposure

Attackers can probe your system with targeted queries to determine whether a specific patient’s data exists in your index. This attack class is now a regulatory concern. Cloud-hosted indexes increase this risk because they expose a remote interface for repeated probing. A locally hosted index removes that external interface and limits access to your internal network.

For example, an attacker sends repeated queries like “Patients diagnosed with HIV in 2024 treated with drug X” and slightly modifies filters each time. They observe changes in response confidence and content. Over time, they infer whether a specific individual’s record exists in your dataset.

These failures show that a BAA does not ensure compliance. An on-premises deployment removes the third-party surface entirely and gives you full control over data flow, access, and auditability.

Split view showing Cloud vs On-premises RAG architecture

What You Are Building

In this section, you will build a RAG system with three layers:

Ingestion layer

You ingest clinical notes and treatment protocols into a controlled vector index with enforced data hygiene. You de-identify data before any processing. HIPAA Safe Harbor requires removal of identifiers, while Expert Determination allows a statistical approach. You apply one of these before ingestion, not after. You then chunk documents into 512 token segments with 50 token overlap, generate embeddings using a local model, and store them in VectorAI DB with metadata.

You define a strict schema for every record. Each chunk includes document type, department, date, and author role. This metadata is not optional. It enables access control at query time and prevents cross-department leakage. Do not store raw documents without structure.

Query layer

You process clinician queries through a controlled retrieval pipeline. Every query passes through role-based access control before it reaches the index. A cardiology user can only retrieve cardiology data. A scheduling bot cannot access diagnosis notes. You enforce this with a MUST filter on department or patient cohort at the database level.

Run hybrid search. Vector similarity retrieves semantically relevant chunks. Metadata filters restrict the result set. Pass the filtered context into a local LLM. The model generates an answer from retrieved data only and includes citations. Do not allow the model to invent or pull from external knowledge.

Audit layer

Log every interaction locally with full traceability. Each query writes a record that includes timestamp, user ID, department, query text, and retrieved document references. This log lives on your infrastructure with defined retention and access policies. You do not rely on external logging systems.

You can reconstruct any access event from this log. You can answer who accessed what, when, and under which role. This satisfies audit requirements and gives your security team direct visibility into system behavior.

The entire system runs on commodity hardware inside the hospital network. The end-to-end architecture of the system is shown in the image:

Hospital RAG system architecture

Building a HIPAA Compliant RAG Workflow

In this section, you will build a fully local RAG system that ingests clinical data, enforces access control, answers queries, and logs every interaction.

Prerequisites

To follow along, install the following tools on your local network:

Docker and Docker Compose installed.
Python 3.10 or higher.
PIP or UV: This guide uses UV.

Step 1: Deploy a vector database

You deploy a local instance of Actian VectorAI DB with persistent storage for both vector data and audit logs.

Create a docker-compose.yaml file:

services:
  vectorai:
    image: williamimoh/actian-vectorai-db:latest
    platform: linux/amd64
    container_name: vectorai_db
    ports:
      - "50052:50051"
    volumes:
      # vector data persists across restarts
      - ./data:/app/data
      # audit log lives on host — not inside the container
      - ./audit_logs:/app/audit_logs
    environment:
      - VECTORAI_LOG_LEVEL=info
    restart: unless-stopped

Run the service:

docker-compose up -d

The database starts and exposes port 50051 for local access. Vector data persists in ./data. Audit logs write to ./audit_logs on the host, which keeps all access records inside your network boundary.

Note:

M3/_M4 Apple Silicons might encounter a GRPC disconnection error without any container _logs. In this case, disable Rosetta in Docker Desktop.
VectorAI DB is under active development.

Step 2: Build the ingestion pipeline

Install the client library and run the ingestion pipeline to convert clinical documents into embeddings and store them in your local vector database.

Use uv for dependency management and execution. It is fast, reproducible, and avoids global Python state.

Download the Actian VectorAI client package. This creates a file actian_vectorai-0.1.0b2-py3-none-any

Initialize your project by:

uv init .
uv venv

After initialization, install the Actian VectorAI package by:

uv pip3 install actian_vectorai-0.1.0b2-py3-none-any

Add the embedding model dependency:

uv add sentence-transformers

Create a file ingest.py with the following contents:

import re
import hashlib
from actian_vectorai import VectorAIClient, Distance, VectorParams, PointStruct
from sentence_transformers import SentenceTransformer

# ── Config ────────────────────────────────────────────────────────────────────
VECTORAI_HOST  = "localhost:50052"
COLLECTION     = "clinical_docs"
EMBED_MODEL    = "sentence-transformers/all-MiniLM-L6-v2"  # 384-dim
VECTOR_DIM     = 384
CHUNK_TOKENS   = 512
OVERLAP_TOKENS = 50

# ── Synthetic clinical notes (replace with real de-identified corpus) ─────────
RAW_NOTES = [
    {
        "document_id":   "card_note_001",
        "document_type": "clinical_note",
        "department":    "cardiology",
        "date":          "2025-03-15",
        "author_role":   "attending_physician",
        "text": """
            Patient: [NAME REDACTED], DOB: [DATE REDACTED], MRN: [MRN REDACTED]
            Chief Complaint: Chest pain radiating to left arm, onset 2 hours ago.
            Assessment: Acute ST-elevation myocardial infarction confirmed on ECG.
            History: Hypertension and type 2 diabetes. Started on aspirin 325 mg,
            clopidogrel 600 mg loading dose, and heparin infusion per ACS protocol.
            Plan: Emergency PCI. Beta-blocker therapy with metoprolol succinate
            25 mg daily post-procedure. ACE inhibitor ramipril 5 mg daily initiated
            24 hours post-PCI. Follow-up echocardiography in 6 weeks.
        """,
    },
    {
        "document_id":   "card_protocol_001",
        "document_type": "treatment_protocol",
        "department":    "cardiology",
        "date":          "2025-01-10",
        "author_role":   "department_head",
        "text": """
            Cardiology Protocol — Heart Failure with Reduced EF (HFrEF)
            First-line therapy:
            - ACE inhibitor: ramipril 2.5–10 mg daily (or ARB if ACE-intolerant).
            - Beta-blocker: bisoprolol 1.25–10 mg daily, carvedilol 3.125–25 mg BID,
              or metoprolol succinate 12.5–200 mg daily. Titrate every 2 weeks.
            - MRA: spironolactone 25–50 mg daily for NYHA class II–IV
              if eGFR > 30 and K+ < 5.0.
            Target: Symptomatic improvement. Reassess LVEF at 3–6 months.
            Device therapy (ICD/CRT) if LVEF ≤ 35% after 3 months optimal therapy.
        """,
    },
    {
        "document_id":   "psych_note_001",
        "document_type": "clinical_note",
        "department":    "psychiatry",
        "date":          "2025-03-18",
        "author_role":   "psychiatrist",
        "text": """
            Psychiatry intake note — [NAME REDACTED], [AGE REDACTED]-year-old.
            Presenting with major depressive episode, PHQ-9 score 18 (severe).
            No current suicidal ideation. Started sertraline 50 mg daily.
            Psychotherapy referral placed. Follow-up in 2 weeks.
            Safety plan documented. Family support confirmed present.
        """,
    },
    {
        "document_id":   "onco_note_001",
        "document_type": "clinical_note",
        "department":    "oncology",
        "date":          "2025-03-20",
        "author_role":   "oncologist",
        "text": """
            Oncology note — [NAME REDACTED].
            Diagnosis: Stage IIIA non-small cell lung cancer, adenocarcinoma.
            EGFR mutation positive (exon 19 deletion).
            Plan: Osimertinib 80 mg daily (first-line EGFR-targeted therapy).
            Baseline CT chest/abdomen/pelvis completed. Brain MRI negative.
            Next imaging review in 8 weeks. Antiemetics PRN, skin care for rash.
        """,
    },
]

# ── Step 1: De-identification ─────────────────────────────────────────────────
# For production use Presidio:
#   from presidio_analyzer import AnalyzerEngine
#   from presidio_anonymizer import AnonymizerEngine
#   analyzer, anonymizer = AnalyzerEngine(), AnonymizerEngine()
#   result = analyzer.analyze(text=raw, entities=[...], language="en")
#   clean  = anonymizer.anonymize(text=raw, analyzer_results=result).text
#
# This demo applies lightweight regex to already-synthetic notes.

_HIPAA_PATTERNS = [
    (r"\b\d{3}-\d{2}-\d{4}\b",                   "[SSN]"),          # SSN
    (r"\bMRN[-:\s]*\d{4,10}\b",                   "[MRN]"),          # medical record #
    (r"\b\d{1,2}/\d{1,2}/\d{2,4}\b",             "[DATE]"),         # dates
    (r"\b[A-Z][a-z]+ [A-Z][a-z]+\b",             "[NAME]"),         # names (simple)
    (r"\b\d{3}[-.\s]\d{3}[-.\s]\d{4}\b",         "[PHONE]"),        # phone
    (r"\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.\w+\b","[EMAIL]"),      # email
    (r"\b\d{5}(?:-\d{4})?\b",                     "[ZIP]"),          # zip
    (r"\b(?:https?://)\S+",                        "[URL]"),          # URLs
    (r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",  "[IP]"),           # IP addresses
]

def deidentify(text: str) -> str:
    """Remove HIPAA Safe Harbor identifiers from text."""
    for pattern, replacement in _HIPAA_PATTERNS:
        text = re.sub(pattern, replacement, text)
    return text.strip()

# ── Step 2: Chunking ──────────────────────────────────────────────────────────
def chunk(text: str, size: int = CHUNK_TOKENS, overlap: int = OVERLAP_TOKENS) -> list[str]:
    """Split text into overlapping token windows (whitespace tokenisation)."""
    tokens = text.split()
    chunks, start = [], 0
    while start < len(tokens):
        end = min(start + size, len(tokens))
        chunks.append(" ".join(tokens[start:end]))
        if end == len(tokens):
            break
        start += size - overlap
    return chunks

# ── Step 3: Embedding ─────────────────────────────────────────────────────────
print(f"Loading embedding model: {EMBED_MODEL}")
_model = SentenceTransformer(EMBED_MODEL)

def embed(texts: list[str]) -> list[list[float]]:
    return _model.encode(texts, normalize_embeddings=True).tolist()

# ── Step 4: Ingest into VectorAI DB ──────────────────────────────────────────
def _chunk_id(doc_id: str, idx: int) -> int:
    """Stable integer ID from (document_id, chunk_index)."""
    h = hashlib.sha256(f"{doc_id}:{idx}".encode()).hexdigest()
    return int(h[:15], 16)

def ingest(notes: list[dict]) -> None:
    with VectorAIClient(VECTORAI_HOST) as client:
        # Health check
        info = client.health_check()
        print(f"VectorAI DB connected  version={info['version']}")

        # Create collection (skip if already exists)
        try:
            client.collections.create(
                name=COLLECTION,
                vectors_config=VectorParams(size=VECTOR_DIM, distance=Distance.Cosine),
            )
            print(f"Collection '{COLLECTION}' created  dim={VECTOR_DIM}")
        except Exception as e:
            if "exists" in str(e).lower():
                print(f"Collection '{COLLECTION}' already exists — skipping create")
            else:
                raise

        total_chunks = 0
        for note in notes:
            # De-identify FIRST — before chunking or embedding
            clean_text = deidentify(note["text"])

            # Chunk second
            chunks = chunk(clean_text)

            # Embed third
            vectors = embed(chunks)

            # Build PointStruct records with strict metadata schema
            # All four metadata fields are REQUIRED — no optional fields.
            points = [
                PointStruct(
                    id=_chunk_id(note["document_id"], i),
                    vector=vectors[i],
                    payload={
                        # ── strict schema ──────────────────────────────────────
                        "document_type": note["document_type"],   # required
                        "department":    note["department"],       # required — RBAC filter key
                        "date":          note["date"],             # required
                        "author_role":   note["author_role"],      # required
                        # ── retrieval helpers ──────────────────────────────────
                        "document_id":   note["document_id"],
                        "chunk_index":   i,
                        "text":          chunks[i],                # de-identified chunk text
                    },
                )
                for i in range(len(chunks))
            ]

            client.points.upsert(COLLECTION, points)
            total_chunks += len(chunks)
            print(f"  ✓ {note['document_id']}  dept={note['department']}  chunks={len(chunks)}")

        print(f"\nIngestion complete — {len(notes)} documents, {total_chunks} chunks total")

if __name__ == "__main__":
    ingest(RAW_NOTES)

This file performs the following actions:

De-identifies data: The system removes all HIPAA identifiers from raw text before processing. Names, dates, and other sensitive fields are replaced with placeholders to prevent PHI from entering the system pipeline.
Chunks the texts: The system splits the cleaned text into 512-token segments with a 50-token overlap. This overlap preserves context across boundaries, enhancing retrieval accuracy.
Embeds the chunks: The model converts each chunk into a numerical vector using a local sentence-transformers model. This process captures semantic meaning while keeping all processing within the network.
Stores with metadata: The system writes each chunk and its vector to VectorAI DB, along with necessary fields like document_type, department, date, and author_role. These fields support strict access control during queries.

Run the script by:

uv run ingest.py

You should see the following results after running the command:

ingest.py execution

From the logs, you see that the ingestion pipeline writes chunks to VectorAI DB.

Step 3: Run your queries

Execute queries against your local RAG system and validate retrieval, access control, and audit logging.

Create a file query.py with the following contents:

import json
import datetime
import urllib.request
import urllib.error
from pathlib import Path
from actian_vectorai import VectorAIClient
from actian_vectorai import FilterBuilder, Field
from sentence_transformers import SentenceTransformer

# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_HOST = "localhost:50052"
COLLECTION    = "clinical_docs"
EMBED_MODEL   = "sentence-transformers/all-MiniLM-L6-v2"
AUDIT_LOG     = Path("./audit_logs/queries.jsonl")   # volume-mounted path

# Ollama settings — set OLLAMA_ENABLED=True once `ollama serve` is running
OLLAMA_ENABLED = False          # flip to True when Ollama is ready
OLLAMA_URL     = "http://localhost:11434/api/generate"
OLLAMA_MODEL   = "mistral"      # or "llama3.2:3b" for lower hardware

# ── RBAC: role → allowed departments ──────────────────────────────────────────
# Access is enforced as a MUST filter at the database level.
# A scheduling_bot cannot reach clinical notes; cardiology cannot see psychiatry.
ROLE_PERMISSIONS = {
    "cardiology_clinician":  ["cardiology"],
    "oncology_clinician":    ["oncology"],
    "general_practitioner":  ["cardiology", "oncology", "general"],
    "admin":                 ["cardiology", "oncology", "psychiatry", "general"],
    "scheduling_bot":        ["scheduling"],   # no clinical note access
}

class AccessDeniedError(Exception):
    pass

def allowed_departments(role: str) -> list[str]:
    if role not in ROLE_PERMISSIONS:
        raise AccessDeniedError(f"Unknown role '{role}' — access denied by default.")
    return ROLE_PERMISSIONS[role]

# ── Embedding (reuse the same model as ingest.py) ─────────────────────────────
print(f"Loading embedding model: {EMBED_MODEL}")
_model = SentenceTransformer(EMBED_MODEL)

def embed(text: str) -> list[float]:
    return _model.encode([text], normalize_embeddings=True).tolist()[0]

# ── Step 5: Search with department MUST filter ─────────────────────────────────
def retrieve(query_vec: list[float], departments: list[str], top_k: int = 5) -> list[dict]:
    """
    Hybrid retrieval: vector similarity + metadata MUST filter.
    Results from departments outside the allowed list are impossible —
    the filter is applied at the database level, not in application code.
    """
    results = []
    with VectorAIClient(VECTORAI_HOST) as client:
        for dept in departments:
            hits = client.points.search(
                collection_name=COLLECTION,
                vector=query_vec,
                limit=top_k,
                # MUST filter — department equality enforced at DB level
                filter=FilterBuilder().must(Field("department").eq(dept)).build(),
            )
            for hit in hits:
                payload = getattr(hit, "payload", {}) or {}
                results.append({
                    "score":         round(getattr(hit, "score", 0.0), 4),
                    "document_id":   payload.get("document_id"),
                    "document_type": payload.get("document_type"),
                    "department":    payload.get("department"),
                    "date":          payload.get("date"),
                    "author_role":   payload.get("author_role"),
                    "chunk_index":   payload.get("chunk_index"),
                    "text":          payload.get("text", ""),
                })

    results.sort(key=lambda r: r["score"], reverse=True)
    return results[:top_k]

# ── Step 6: LLM answer via Ollama ─────────────────────────────────────────────
_RAG_SYSTEM = (
    "You are a clinical decision support assistant. "
    "Answer ONLY using the context passages below. "
    "Do NOT use external knowledge or make assumptions. "
    "Cite each fact as [Doc N]. "
    "If the context is insufficient, say: 'I cannot answer from the available documents.'"
)

def build_context(chunks: list[dict]) -> str:
    return "\n\n".join(
        f"[Doc {i+1}] ({c['document_type']}, dept={c['department']}, "
        f"date={c['date']}, role={c['author_role']})\n{c['text']}"
        for i, c in enumerate(chunks)
    )

def generate(query_text: str, chunks: list[dict]) -> str:
    if not OLLAMA_ENABLED:
        # Return raw retrieved context when LLM is disabled
        return "[LLM disabled — set OLLAMA_ENABLED=True]\n\n" + build_context(chunks)

    context = build_context(chunks)
    prompt = f"{_RAG_SYSTEM}\n\nContext:\n{context}\n\nQuestion: {query_text}\n\nAnswer:"

    payload = json.dumps({
        "model":  OLLAMA_MODEL,
        "prompt": prompt,
        "stream": False,
        "options": {"num_predict": 400},
    }).encode()

    try:
        req = urllib.request.Request(
            OLLAMA_URL,
            data=payload,
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        with urllib.request.urlopen(req, timeout=30) as resp:
            data = json.loads(resp.read())
            return data.get("response", "").strip()
    except urllib.error.URLError as e:
        return f"[Ollama unreachable: {e}]\n\nRetrieved context:\n{build_context(chunks)}"

def write_audit(record: dict) -> None:
    AUDIT_LOG.parent.mkdir(parents=True, exist_ok=True)
    with open(AUDIT_LOG, "a", encoding="utf-8") as f:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

# ── Public query entry point ───────────────────────────────────────────────────
def query(user_id: str, role: str, query_text: str, top_k: int = 5) -> dict:
    """
    Execute a role-gated RAG query.

    Returns:
        {answer, retrieved_docs, access_denied, error}
    """
    timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat()

    # RBAC check — before anything else
    try:
        departments = allowed_departments(role)
    except AccessDeniedError as e:
        write_audit({
            "timestamp": timestamp, "user_id": user_id, "role": role,
            "department": "DENIED", "query_text": query_text,
            "retrieved_docs": [], "answer_provided": False, "access_denied": True,
            "denial_reason": str(e),
        })
        return {"answer": f"Access denied: {e}", "retrieved_docs": [], "access_denied": True}

    # Embed → retrieve (with MUST filter) → generate
    q_vec  = embed(query_text)
    chunks = retrieve(q_vec, departments, top_k)
    answer = generate(query_text, chunks)

    doc_refs = [
        {"document_id": c["document_id"], "chunk_index": c["chunk_index"],
         "department": c["department"], "document_type": c["document_type"],
         "score": c["score"]}
        for c in chunks
    ]

    # Audit log — every query, regardless of outcome
    write_audit({
        "timestamp":       timestamp,
        "user_id":         user_id,
        "role":            role,
        "department":      ",".join(departments),
        "query_text":      query_text,
        "retrieved_docs":  doc_refs,
        "answer_provided": True,
        "access_denied":   False,
    })

    return {"answer": answer, "retrieved_docs": doc_refs, "access_denied": False}

# ── Demo runs ─────────────────────────────────────────────────────────────────
if __name__ == "__main__":

    separator = "─" * 60

    # ── Query 1: authorised cardiology query ────────────────────────────────
    print(f"\n{separator}")
    print("QUERY 1 — cardiology_clinician (authorised)")
    print(separator)
    r1 = query(
        user_id="dr_chen_007",
        role="cardiology_clinician",
        query_text="What beta-blocker is recommended for heart failure with reduced ejection fraction?",
    )
    print(f"\nAnswer:\n{r1['answer']}\n")
    print("Retrieved sources:")
    for d in r1["retrieved_docs"]:
        print(f"  score={d['score']}  [{d['document_type']}]  dept={d['department']}  "
              f"doc={d['document_id']}  chunk={d['chunk_index']}")

    # ── Query 2: scheduling bot tries to access clinical notes ───────────────
    print(f"\n{separator}")
    print("QUERY 2 — scheduling_bot (attempting clinical note access)")
    print(separator)
    r2 = query(
        user_id="bot_sched_01",
        role="scheduling_bot",
        query_text="What are the diagnosis notes for cardiology patients?",
    )
    if r2["access_denied"]:
        print(f"\n✗ Access denied (as expected): {r2['answer']}")
    else:
        print(f"\nAnswer:\n{r2['answer']}")
        print("Sources:", r2["retrieved_docs"])

    # ── Query 3: cardiology query that must NOT return psychiatry notes ───────
    print(f"\n{separator}")
    print("QUERY 3 — cardiology_clinician (RBAC must exclude psychiatry)")
    print(separator)
    r3 = query(
        user_id="dr_chen_007",
        role="cardiology_clinician",
        query_text="antidepressant dosing and patient management",
    )
    departments_returned = {d["department"] for d in r3["retrieved_docs"]}
    cross_leak = "psychiatry" in departments_returned
    print(f"\nDepartments in results: {departments_returned or 'none'}")
    print(f"Cross-department leak: {'✗ LEAK DETECTED' if cross_leak else '✓ none — RBAC working correctly'}")

    # ── Show audit log tail ───────────────────────────────────────────────────
    print(f"\n{separator}")
    print("AUDIT LOG  →  {AUDIT_LOG}")
    print(separator)
    if AUDIT_LOG.exists():
        lines = AUDIT_LOG.read_text().strip().splitlines()
        for line in lines[-3:]:          # show last 3 entries
            entry = json.loads(line)
            print(json.dumps({
                "timestamp":     entry["timestamp"],
                "user_id":       entry["user_id"],
                "role":          entry["role"],
                "query_text":    entry["query_text"][:60] + "…",
                "docs_accessed": len(entry["retrieved_docs"]),
                "access_denied": entry["access_denied"],
            }, indent=2))
    else:
        print("No audit log found — run ingest.py first.")

The script performs three core operations in a single flow.

Enforces access control: The system checks the user’s role before retrieving any data. Each role is mapped to specific departments, and enforced as a mandatory filter at the database level. The authorization layer immediately blocks and logs unauthorized roles.
Retrieve and generate answers: The system embeds the query and retrieves relevant document chunks using vector search, with strict department filters applied. The results are then passed to a local LLM. If the LLM is disabled, the retrieved context is returned directly.
Write audit logs: The system logs every query locally, including the user ID, role, query text, accessed documents, and access status. This creates a complete audit trail for compliance and review.

Run the script by:

uv run query.py

You should see the following results after running the command:

query.py execution

The output shows three test cases: one showing a valid clinician query, a denied access attempt, and a check for cross-department leakage. These confirm that RBAC and audit logging work correctly before you move to production.

Step 4: Configure the audit log

Store every query locally by using the volume mapping defined during deployment.

The Docker configuration mounts ./audit_logs from your host into the container. When you run queries, this creates a local folder named audit_logs with a file queries.jsonl.

The file contains the following entries:

{"timestamp": "2026-03-31T15:56:02.552088+00:00", "user_id": "dr_chen_007", "role": "cardiology_clinician", "department": "cardiology", "query_text": "What beta-blocker is recommended for heart failure with reduced ejection fraction?", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

{"timestamp": "2026-03-31T15:56:03.098346+00:00", "user_id": "bot_sched_01", "role": "scheduling_bot", "department": "scheduling", "query_text": "What are the diagnosis notes for cardiology patients?", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

{"timestamp": "2026-03-31T15:56:03.663767+00:00", "user_id": "dr_chen_007", "role": "cardiology_clinician", "department": "cardiology", "query_text": "antidepressant dosing and patient management", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

{"timestamp": "2026-03-31T15:56:18.331657+00:00", "user_id": "dr_chen_007", "role": "cardiology_clinician", "department": "cardiology", "query_text": "What beta-blocker is recommended for heart failure with reduced ejection fraction?", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

{"timestamp": "2026-03-31T15:56:18.492188+00:00", "user_id": "bot_sched_01", "role": "scheduling_bot", "department": "scheduling", "query_text": "What are the diagnosis notes for cardiology patients?", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

{"timestamp": "2026-03-31T15:56:18.569824+00:00", "user_id": "dr_chen_007", "role": "cardiology_clinician", "department": "cardiology", "query_text": "antidepressant dosing and patient management", "retrieved_docs": [], "answer_provided": true, "access_denied": false}

Each line represents a single query event. The log captures who made the request, their role, the department scope, the query text, and whether access was allowed. This file lives entirely on your infrastructure and gives you a complete, verifiable audit trail for every interaction with PHI.

Wrapping Up

A BAA was never enough because it does not control how your application handles PHI. You solved that by keeping all data, queries, and logs inside your network.

You now have a RAG system that enforces role-based access, retrieves only authorized data, and logs every interaction locally, without external APIs or third-party exposure.

Apply this pattern to other regulated systems. Refer to the VectorAI DB documentation and GitHub repository for updates and implementation details.

Join the community and learn more about Actian.

When to choose on-premises vs. cloud for vector databases

Offisong Emmanuel — Tue, 19 May 2026 19:42:27 +0000

For most of the last decade, the on-premises vs. cloud debate felt settled. Cloud computing was cheaper, faster, and easier to adopt. Enterprises moved workloads from on-premises infrastructure to public cloud services, relying on major cloud providers to handle scalability, maintenance, and security.

In 2026, that assumption is breaking, and cracks are showing up in legal reviews, financial projects, and SLA negotiations. Enterprises are facing an increasing pressure for data residency regulations, stricter enforcement, and scrutiny around cloud security models. Compliance constraints, data security requirements, cost predictability, and latency are forcing teams to reconsider on-premises solutions, private cloud computing, and hybrid cloud infrastructure.

At the same time, AI is moving closer to where data is generated. Manufacturing sites, retail stores, and healthcare environments increasingly require offline capability and sub-100ms latency. That shift helps explain why Oracle released AI Database 26ai for on-premises deployment and why Google is pushing Gemini onto Distributed Cloud for air-gapped environments. This shift signals that large-scale enterprise AI no longer fits neatly in cloud environments.

In this article, we’ll examine why on-premises infrastructure is resurging, what trade-offs you need to know, and how to make defensible deployment decisions.

What’s Driving The On-premises Resurgence

The renewed interest in on-premises infrastructure is not about going back to old systems. It is a response to clear changes in how AI systems are being built and used in 2025 and 2026. For many enterprises, cloud-only vector databases no longer fit their compliance, cost, and reliability needs.

A lot of factors drive this current on-premises resurgence, but in this article we will consider four key causes.

Large vendors now support on-premises AI

On-premises AI is no longer treated as an edge case by major vendors. Oracle’s release of AI Database 26ai and Google’s decision to run Gemini on Distributed Cloud show a clear shift in how enterprise AI is being packaged and delivered.

These products are built for large enterprises, not early-stage experiments or research projects. That distinction matters. Large vendors do not invest in complex on-premises AI platforms unless there is strong and growing customer demand. These announcements confirm that many enterprises want to run AI systems inside their own environments, close to their data, and under their full operational control. Why is this?

Regulatory pressure is now a real blocker

Teams used to plan for regulatory risk as a future possibility. Now it’s a day-to-day reality. GDPR enforcement reached record levels in 2025, with insufficient legal basis for data processing driving the largest penalties. That year alone, regulators issued nearly 2,700 fines totaling billions of euros.

From a data security perspective, GDPR enforcement has fundamentally changed how enterprises evaluate cloud services. While cloud service providers offer compliance tooling, legal teams are increasingly wary of relying on third-party providers for sensitive data storage and processing.

HIPAA adds another layer of complexity. For example, in Florida, physicians must maintain medical records for five years after the last patient contact, whereas hospitals must maintain them for seven years under state record-retention requirements. This makes repeated data movement risky and expensive. Financial services and government contractors face similar data sovereignty requirements that limit where data can be stored and processed. In these situations, cloud deployments add legal review, audit work, and ongoing risk. Keeping data on-premises is often the most straightforward way to meet these obligations.

Edge AI requires local and offline operation

AI workloads are increasingly deployed close to where data is created. Manufacturing facilities may operate in air-gapped environments or remote locations with limited connectivity. Retail systems must continue working during network outages. Healthcare applications often require very low latency for real-time decision support.

In these environments, relying on a remote cloud service introduces risk. Network delays and outages directly affect system reliability. On-premises and edge deployments allow vector search and inference to run locally, without depending on constant network access. For many use cases, this local execution is not an optimization but a requirement.

Together, these shifts explain why on-premises vector databases are gaining traction again. The change is driven by the practical realities of deploying production AI systems under real regulatory, cost, and reliability constraints.

The Compliance Calculus

For many enterprises, compliance is the deciding factor in the on-premises versus cloud debate. While cloud providers offer compliance certifications, the real challenge is not whether a platform can be compliant in theory, but whether it can withstand legal review, audits, and long-term operational scrutiny in practice. Once vector databases move into production and begin storing sensitive or regulated data, these questions become unavoidable.

GDPR and the limits of cross-border transfers

The Schrems II ruling changed how European data can be processed outside the EU. Privacy Shield was invalidated, leaving Standard Contractual Clauses as the primary legal mechanism for cross-border data transfers. In highly regulated industries such as financial services and healthcare, many legal teams consider SCCs insufficient due to enforcement uncertainty and ongoing legal challenges.

For vector databases, this matters because embeddings often contain derived personal data. Even if raw records are masked or tokenized, embeddings can still be considered personal data under GDPR. If data must remain within the EEA, or within a specific country, cloud deployments that rely on global infrastructure introduce legal risk. In these cases, on-premises or in-region deployment becomes a requirement rather than a preference.

HIPAA retention and the real cost of data movement

HIPAA does not explicitly require data to stay on-premises, but it does require long retention periods and strict access controls. When vector embeddings are built on top of this data, they inherit the same retention requirements. HIPAA data governance must be enforced when considering on-premises or cloud vector databases.

The cost impact becomes clear when egress fees are included. Consider a system storing 100 TB of embeddings in a cloud environment. At a common egress rate of $0.09 per GB, moving that data out of the cloud over a seven-year retention period results in:

100 TB × $0.09 per GB × 84 months = over $750,000 in egress costs alone

This does not include compute, storage, or indexing costs. With this in mind, will cloud data warehouses really help you cut costs?

Financial services and data sovereignty rules

Financial institutions face additional constraints beyond GDPR. Regulations such as GLBA, APRA, and regional data sovereignty mandates often require strict control over where customer data is stored and processed. Regulators may demand clear evidence of geographic boundaries, access controls, and auditability.

Cloud services can meet some of these requirements, but they often introduce complex configurations, contractual dependencies, and ongoing compliance reviews. For many banks and insurers, on-premises deployment simplifies audits by keeping data within controlled infrastructure that regulators already understand.

Government and public sector constraints

Government contracts introduce some of the strictest infrastructure requirements. Standards such as FedRAMP often mandate US-only infrastructure, restricted access, and tightly controlled environments.

In these cases, public cloud services are frequently disallowed or require extensive approvals. On-premises deployment is often the only viable option for running vector databases in support of government workloads.

When compliance makes cloud untenable

If legal teams flag cross-border data transfers as unacceptable, cloud deployments quickly become impractical. Once data residency is mandatory, on-premises deployment is no longer a trade-off decision. It is a compliance requirement.

The Cost Breakdown Analysis

Cost is often the reason teams revisit the on-premises versus cloud decision. To make a defensible decision, teams need to understand where costs diverge and when self-hosting becomes economically rational.

Where self-hosting breaks even

Research from OpenMetal shows a consistent breakeven point for Pinecone vector databases at scale. Once workloads reach roughly 80 to 100 million queries per month, self-hosted deployments tend to be cheaper than managed cloud services. Below this range, cloud pricing is usually competitive. Above it, usage-based billing begins to dominate total cost.

This threshold matters because many enterprise RAG systems cross it quickly. Customer support, document search, fraud detection, and recommendation systems often serve tens or hundreds of millions of queries each month once deployed across business units or regions.

The hidden cost in cloud pricing

Cloud pricing is rarely just a per-query fee. Vector databases introduce several cost drivers that are easy to overlook during planning.

Egress fees are a major factor. Most cloud providers charge around $0.09 per GB for data leaving their network. Moving embeddings between regions, exporting data for analytics, or migrating to another system all incur these fees. Over time, they become a meaningful portion of total spend.

Finally, vector search does not scale linearly. As vector counts grow and dimensionality increases, query costs rise faster than expected. What looks affordable at 10 million vectors can become expensive at 500 million, even if query volume grows steadily.

On-premises costs are fixed and predictable

On-premises deployments have real costs, but they behave differently. Hardware is typically amortized over three to five years. Staffing requirements are stable once the system is running. Facilities and power costs are known in advance.

The key difference is predictability. Costs do not spike because of usage patterns or data movement. Once the system is sized correctly, monthly spend remains largely flat, even as query volume increases.

A real world example

Consider a production e-commerce application with the following scale:

500M vectors
200M queries every month
1024 vector dimensions
6M writes monthly

At this scale, a typical managed Pinecone vector database costs around $8,500 per month once compute, storage, and rebuild overhead are included.

Estimated monthly cost

Total Estimated Cost: $8,454 / month

1. Storage

Usage: 845 GB
Cost: $279

2. Query Costs

Configuration:
24 b1 nodes
4 shards × 6 replicas
Assumption: 1% filter selectivity
Estimated Cost: $8,074
Note: Actual query cost may vary. Benchmark your workload on DRN for more accurate estimates.

3. Write Costs

Write Volume: 30 million Write Units (WU)
Assumption: Each write request consumes ≥ 5 WU
Cost: $101

An equivalent on-premises deployment might cost approximately half of that after hardware amortization, assuming an 18-month payback period and one to two engineers supporting the system. After that payback period, costs drop further while capacity remains available.

A study by Enterprise Storage Forum shows the cost projection of on-premises and cloud workloads.

Cost alone does not decide every deployment, but once vector workloads reach scale, the economics become difficult to ignore. Understanding where your system sits on this curve is essential before locking in a long-term vector database strategy.

When Latency And Connectivity Matter

Latency and connectivity are often treated as secondary concerns in architecture decisions. For many AI workloads, they are decisive. Once vector databases support real-time systems, network round-trips and internet dependency can make cloud deployments impractical or unsafe.

Real-time response requirements

Some applications have strict response time limits. In healthcare, clinical decision support and diagnostic systems often require responses in under 50 milliseconds. This budget includes data retrieval, vector search, and model inference. Similarly, banks and financial institutions often require very low latency for maximum user experience.

Public cloud deployments add unavoidable network latency. Even within the same region, round-trip latency typically adds 20 to 80 milliseconds before any compute work begins. For applications with tight latency targets, this overhead alone can exceed the total allowed response time. On-premises deployments remove that network hop, allowing systems to meet real-time requirements consistently.

Systems that must work offline

Many environments cannot rely on constant connectivity. Retail point-of-sale systems must continue operating during network outages. Manufacturing facilities are often located in remote areas with unstable connections. Military and maritime deployments may operate in fully disconnected or classified environments.

In these scenarios, a cloud dependency is a single point of failure. If the network goes down, the AI system stops working. On-premises and edge deployments allow vector search and inference to run locally, ensuring the system continues to function even when external connectivity is unavailable.

The cost of downtime

It is no news that there has been an increase in downtime from cloud providers. On November 18, 2025, Cloudflare outage disrupted large portions of the internet, causing downtime across major platformsincluding X, Amazon Web Services, Spotify, and so on. The impact of connectivity failures is not theoretical. In manufacturing, average downtime costs are estimated at $260,000 per hour. When AI systems support quality control, predictive maintenance, or process automation, any outage directly affects production.

A cloud-only architecture introduces risk that is hard to justify in these environments. Even short network disruptions can lead to significant financial loss. On-premises deployments reduce this risk by removing external dependencies from critical execution paths.

For workloads with strict latency targets or limited connectivity, the choice is often clear. Cloud-based vector databases may work during development, but they fail to meet operational requirements in production.

The Operational Complexity Question

The strongest argument for cloud vector databases is operational simplicity. Managed services remove the need to provision hardware, manage clusters, apply patches, or handle failures. For small teams or early-stage projects, this advantage is real and often decisive. Cloud deployments allow engineers to focus on application logic rather than infrastructure.

It is also important to recognize that modern on-premises deployments look very different from those of a decade ago. This is not the world of manual server provisioning and fragile scripts. Kubernetes, infrastructure-as-code, and automated deployment pipelines have reduced operational overhead significantly. Rolling upgrades, automated scaling, and monitoring are now standard practices in on-premises environments as well as in the cloud.

Many enterprises adopt hybrid approaches to balance speed and control. Development and experimentation happen in the cloud, where teams can move quickly and iterate. Production systems run on-premises, where costs are predictable and compliance is easier to enforce. This pattern allows teams to get the best of both models without committing fully to either.

Decision Framework: Eight Questions

The fastest way to make a defensible deployment decision is to walk through a small set of yes or no questions with engineering, legal, finance, and operations.

1. Does your data require geographic restrictions?

Regulations such as GDPR, HIPAA, and financial services rules may limit where data can be stored or processed.

If yes, on-premises should be strongly considered because it provides full control over data location. If no, cloud deployment remains viable.

2. Do you have predictable, high-volume query patterns?

Cloud vector database costs scale with usage. A simple check is monthly queries multiplied by the unit cost.

If usage exceeds roughly 80 to 100 million queries per month, on-premises is often cheaper. Below that range, cloud pricing is usually more economical.

3. Do you need offline capability?

Some systems must continue working without network access, such as in manufacturing, retail, or edge environments.

If yes, on-premises is required. If no, cloud remains an option.

4. Can you tolerate additional latency?

Cloud deployments add network latency, often 50 to 100 milliseconds.

If your application cannot tolerate this, on-premises deployment is necessary. If it can, cloud performance may be acceptable.

5. Do you have existing infrastructure teams?

Operational capacity matters.

If you already run on-premises systems, the added burden is limited. If not, cloud-managed services provide a clear operational advantage.

6. Is cost predictability important?

Usage-based billing introduces cost variability.

If predictable costs matter, on-premises provides stability. If flexibility matters more, cloud pricing may be a better fit.

7. Are you extending existing IT infrastructure?

Deployment context affects the decision.

If you are extending existing systems, on-premises leverages current investments. If you are building something new, cloud may be faster to deploy.

8. How large is your data footprint?

Data volume and access frequency influence long-term cost.

If you manage more than 10 TB with frequent access, on-premises becomes attractive. If your data is smaller, cloud is often sufficient.

When several answers point in the same direction, the decision becomes easy to explain and defend across engineering, legal, finance, and operations teams.

When Cloud Makes Sense

On-premises deployment is not always the right answer. In many situations, cloud-based vector databases remain the better choice. Being clear about these cases helps avoid over-engineering.

Unpredictable scaling: Startups and new products often face uncertain growth. Cloud platforms allow rapid scaling without long-term infrastructure commitments, which reduces risk when demand is unclear.
Small data volumes: When total data is under 10 TB and query volume stays below about 50 million queries per month, cloud pricing usually works well and is simpler than self-hosting.
Rapid experimentation: Proof-of-concepts, research projects, and early prototypes benefit from fast setup and easy teardown. Cloud services support quick iteration with minimal operational effort.
No compliance constraints: If data residency, sovereignty, and regulatory requirements are not an issue, cloud deployment avoids legal complexity and speeds up delivery.
Limited infrastructure expertise: Teams focused on application logic rather than operations can rely on managed services instead of maintaining databases, clusters, and hardware.

In these cases, cloud is the most effective and practical option.

Hybrid Deployment Strategies

Hybrid deployments act as the middle ground for enterprises that need both speed and control. Rather than treating cloud and on-premises as mutually exclusive, teams place each part of the system where it performs best.

Cloud for iteration, on-prem for scale

A common pattern is to develop and test in the cloud, where managed services and elastic infrastructure enable rapid iteration. Once models, indexes, or pipelines are stable, they are promoted into on-premises production environments to meet compliance, latency, and operational requirements. This preserves developer velocity without compromising production guarantees.

Data segregation by risk and regulation

Hybrid architectures also allow organizations to separate workloads by risk profile. Sensitive or regulated data stays on-premises, while analytics, training, or search over derived data runs in the cloud. The same logic applies regionally: EU data may remain on-premises or in sovereign environments, while US workloads run in public cloud regions, avoiding global systems being constrained by the strictest jurisdiction.

Cost and migration flexibility

Cost optimization is another driver. Frequently accessed vectors or low-latency services can be cheaper and more predictable on-premises, while cold storage and bursty workloads benefit from cloud pricing. Many teams start cloud-first, then selectively move components on-premises as scale or compliance pressures grow. Hybrid makes this a controlled evolution rather than a disruptive rewrite.

Industry research shows this is a stable operating model. Google Distributed Cloud and similar platforms explicitly frame hybrid as a long-term strategy, recognizing that modern systems are designed to span environments, not collapse them into one.

Actian’s Approach To On-premises Vector Databases

For teams that conclude that on-premises is the right deployment model, the next question is: which platform can actually meet these requirements? Actian’s approach is built specifically for this audience, without assuming the cloud is the default or the end state.

Actian delivers an enterprise-grade vector database that runs fully in your own data center or controlled environments. You retain full control over data placement, networking, and operations. There is no forced dependency on external cloud services, which simplifies audits and long-term system design.

Compliance requirements are treated as baseline constraints. By keeping data local and eliminating egress paths, Actian aligns with GDPR, HIPAA, FedRAMP, and similar regulatory frameworks. This reduces the need for compensating controls or complex legal workarounds.

Cost behavior is also predictable. Actian avoids usage-based pricing models that scale with queries or vector counts. This makes budgeting simpler and removes surprises as workloads grow.

Edge support is also taken into consideration. Actian’s architecture supports offline operation and local inference, making it suitable for manufacturing sites, retail locations, and other environments where connectivity is limited or unreliable. The system is designed to keep working even when the network does not.

Final Thoughts

Choosing between cloud and on-premises for vector databases is about understanding your priorities. Cloud works well for small workloads, rapid experimentation, and teams without deep infrastructure expertise. On-premises makes sense when compliance, latency, cost predictability, or scale are critical.

Many enterprises find a hybrid approach is the best balance, combining cloud flexibility with on-premises control. The key is making intentional decisions based on your data, workloads, and regulatory needs rather than following trends.

Actian empowers enterprises to confidently manage and govern data at scale. Organizations trust Actian data management and data intelligence solutions to streamline complex data environments and accelerate the delivery of AI-ready data. As the data and AI division of HCLSoftware, Actian helps enterprises manage and govern data at scale across on-premises, cloud, and hybrid environments. Learn more about Actian and how it fits into your on-premises AI strategy.

How to Evaluate Vector Databases in 2026

Odewole Babatunde Samson — Tue, 19 May 2026 09:12:21 +0000

In 2026, a synthetic performance crisis challenges the vector database market. A GitHub search for “vector database benchmark” reveals polished repositories with dashboards and performance charts. However, vendors often build these tools to evaluate their own products and portray architecture-specific strengths as objective comparisons.

Zilliz maintains VectorDBBench. Redis and Qdrant publish benchmark suites that highlight their own systems. Even widely cited Approximate Nearest Neighbor (ANN) evaluations, such as ANN-Benchmarks, rely on low-dimensional datasets such as Scale-Invariant Feature Transform (SIFT) and Generalized Search Trees (GIST). Modern Large Language Model (LLM) embeddings often reach 3,072 dimensions. These benchmarks do not reflect that reality.

Leaderboards reward performance under static conditions, yet production systems must survive continuous writes, metadata filters, and concurrency spikes. As software engineer Simon Frey famously noted in a viral post: “The best vector database is the one you already have.” This captures the 2026 market shift, prompting teams to move from specialized silos toward the databases they already trust and operate.

This guide takes a production-first approach. We define the five critical tests for 2026 and explore why your optimal vector database may already exist within your current architecture, whether that is PostgreSQL with pgvector or an enterprise hybrid engine like Actian VectorAI DB.

TL;DR

The bias: Most benchmark suites originate from vendors and optimize for narrow architectural advantages.
The reality: Production workloads include continuous ingestion, metadata filtering, and concurrency spikes that synthetic tests ignore.
The risk: Tail latency (P99), index fragmentation, and write amplification degrade systems long before average QPS drops.
The cost curve: Managed vector services often introduce nonlinear pricing as the dataset size increases.
The direction: 2026 favors integrated platforms, from established relational extensions (PostgreSQL + pgvector) to enterprise hybrid systems (Actian VectorAI DB), over “vector-only” silos.

Why Every Benchmark You’ve Seen is Vendor-Optimized

Benchmarks create a perception of objectivity but often encode architectural assumptions. Tools like VectorDBBench (Zilliz) reward distributed scaling, while Redis and Qdrant suites emphasize in-memory operations. To find objective data, architects must look to peer-reviewed academic conferences such as NeurIPS and VLDB (Very Large Databases), which prioritize algorithmic rigor over marketing.

Before examining what matters in production, it helps to understand how common benchmark tools shape outcomes.

Benchmark tool	Primary creator	Optimization focus	Typical bias
VectorDBBench	Zilliz (Milvus)	High-throughput scaling	Favors massive clusters; penalizes single-node systems.
vector-db-benchmark	Redis/Qdrant	In-memory operations	Favors RAM-heavy architectures; ignores TCO of memory.
ANN-Benchmarks	Academic	Raw algorithm efficiency	Uses outdated, low-dimensional datasets (SIFT/GIST).
NeurIPS / VLDB	Academic Peers	Algorithmic robustness	Focuses on math/theory; ignores operational/SLA reality.

The Hidden Rules of Benchmarking

A significant hurdle is the “DeWitt Clause,” a legal provision in many End User License Agreements (EULAs) that prohibits users from publishing independent benchmarks without the vendor’s permission. In 2024, BenchANT found that 30% of the major vector databases legally prohibit disclosure that their products are slow.

Furthermore, these benchmarks often operate at “Time Zero,” the artificial window immediately following ingestion but preceding live updates. In production, systems must constantly insert and delete data, forcing the index to re-optimize in real time. Vendor benchmarks often omit the Out-of-Memory (OOM) failures that result.

The Five Production Tests That Actually Matter

Most benchmarks measure performance after loading data, before any real updates occur. But production is a nonstop, unpredictable process. To find a database that can handle real users, you should run these five stress tests.

1. Filtering under concurrent load

Pure vector similarity searches are rare in real life. In production, you’re more likely to search for something like “Product recommendations WHERE category is ‘shoes’ AND stock > 0.”

Reddit’s engineering team, managing 340M+ vectors, identified metadata filtering as the primary performance bottleneck in their 2025 deployment. They found that as concurrent users grew, the database spent more time resolving metadata filters than calculating similarity distances.

The reality: Production means 100+ concurrent clients hitting different metadata subsets.
The gap: VectorDBBench only tests with a single client. In real-world situations, moving data between the vector graph and the relational metadata store can cause P99 latency to jump by 10x, as the CPU waits for disk I/O.

2. Performance degradation over time

While archival retrieval-augmented generation (RAG) systems can technically use static knowledge bases, production-grade applications in 2026 must reflect real-time data, such as customer tickets or product inventory. As the engineering team at Milvus admitted, “Benchmarks test after data ingestion completes, but production data never stops flowing.” If the database cannot re-index as quickly as it ingests data, your AI may provide stale or incorrect answers for hours.

Benchmarks that omit a “72-hour continuous write-and-query” test provide zero value. You must determine whether query performance degrades after six months of continuous index maintenance.

3. Tail latency under load (P95/P99)

Average latency can be misleading and doesn’t show what users really experience. For example, a 10ms average response time doesn’t help if your slowest 1% of queries (P99) take 800ms. This makes your AI agent seem slow and unreliable. Only high-concurrency tests reveal these spikes, which often happen during garbage collection or index locking.

4. Total cost of ownership (TCO)

In 2025, managed vendors introduced complex “read unit” pricing. This created a “Growth penalty”: if your index grows from 10GB to 100GB, you may pay 10x as much for the same query result.

Scale metric	Managed Vector DB (usage-based)	Integrated/Hybrid platform	TCO impact
Initial (10GB)	High (Platform fee + usage)	Moderate (Fixed resource)	Integrated is ~40% lower
Growth (100GB)	High (Scales with volume)	Low (Vertical scaling)	8x cost gap
Enterprise (1TB+)	Prohibitive (Linear growth)	Optimized (Reserved capacity)	90%+ long-term savings

This economic reality primarily drives the market’s shift toward “Vector as a Feature,” in which teams prioritize on-premises capabilities and predictable scaling over usage-based silos.

5. Operational maturity

Benchmarks ignore the “Operational Support Tax,” which quantifies the cost and risk of maintaining specialized infrastructure. You can easily find a PostgreSQL expert because the community has thrived for 30 years, but hiring someone proficient in a niche, three-year-old vector database often creates a bottleneck.

Evaluate the ecosystem: Does the database work with standard backup tools? Can it integrate with Prometheus? How long does it take to rebuild an index after a crash?

Here’s how benchmark claims compare to production reality.

Metric	Benchmark focus	Production reality
Ingestion	Static QPS after completion	Sustained QPS during continuous writes
Latency	Average latency	P95/P99 Latency under concurrent load
Filtering	Single-client filtered search	100+ Concurrent metadata-filtered queries
Cost	Infrastructure cost per query	TCO at 100M+ queries/month

Spotting these hidden bottlenecks is the first step to building a strong system. In 2026, the answer is rarely to use a faster, specialized database. Instead, engineers are adding these features to the tools they already know and trust.

The Consolidation Shift: Vector as a Feature

Corey Quinn, Chief Cloud Economist, once said: “Vector is a feature, not a product.” This prediction shapes the 2026 market. Teams are moving away from specialized “Vector-Only” databases and choosing integrated “Vector-Also” platforms. Shifting data between a main database and a separate vector database often causes more problems than it fixes.

The PostgreSQL renaissance

Engineers frequently argue on platforms like Hacker News that ~80% of RAG use cases (specifically those with embeddings under 2M) do not require a specialized vector database. For these workloads, standalone silos often introduce more operational friction than they offer in performance gains. Instacart validated this at scale by migrating from Elasticsearch to PostgreSQL, achieving 80% cost savings and reducing write workload by 10x after eliminating the need to coordinate and reconcile data across fragmented architectures.

Recently, pgvectorscale achieved 471 queries per second at 99% recall on 50 million vectors, outperforming Qdrant’s 41 QPS on identical AWS hardware. Vendor benchmarks often omit this result because it shows that most RAG applications don’t require a specialized vendor.

Performance metric	PostgreSQL (pgvector + pgvectorscale)	Qdrant (Specialized)	The Delta
Throughput (QPS)	471.57	41.47	11.4x higher in Postgres
P95 Latency	60.42 ms	36.73 ms	Qdrant is 39% faster at tail
P99 Latency	74.60 ms	38.71 ms	Qdrant is 48% faster at tail
Hardware	AWS r6id.4xlarge (16 vCPU)	AWS r6id.4xlarge (16 vCPU)	Parity

The integrated enterprise gap

For workloads that exceed basic extensions, Actian VectorAI DB bridges the gap by embedding a high-performance engine with native vector support. Teams can execute metadata filtering and similarity search within a single system, reducing data movement and simplifying query execution.

Platform	Architectural strategy	Intended AI capability
Actian VectorAI DB	High-performance hybrid	Engineered for integrated analytics + native vector support.
PostgreSQL	Integrated feature	Leverages pgvector within standard SQL.
AWS S3 Vectors	Storage-centric	Designed to query multi-billion vectors in object storage.
MongoDB Atlas	Unified document/vector API	Integrates native vector search directly into the existing document store workflow.

As the market comes together, the way we evaluate databases shifts. Teams no longer ask, “Who has the fastest graph?” They ask, “Which architecture provides the most reliable query engine?” No universal winner exists. Teams instead face a spectrum of trade-offs between specialized speed and integrated reliability.

The evaluation process now puts more weight on operational strength, real-world flexibility, and support for hybrid search. Reliable query execution is becoming the top priority, especially given the growing demand for hybrid search.

Hybrid Search Reality That Pure Vector Benchmarks Hide

Pure vector search often fails the “groundedness” test, which measures how strictly an AI’s response relies on provided source material. A high groundedness score ensures that the LLM avoids fabrication and adheres closely to your internal data.

According to an analysis by the Microsoft Azure DevBlog, pure vector search alone struggles with factual accuracy, scoring a mediocre 2.79 out of 5 for groundedness. The solution is Hybrid Search, which blends semantic vector similarity with traditional keyword matching (BM25).

The 20–40% performance penalty

Hybrid search demands significant computation. The database must rank results from two different engines, such as lexical and semantic, then merge them using a fusion algorithm. Production implementations typically see a 20–40% performance penalty when moving from pure vector search to hybrid search. Reciprocal Rank Fusion (RRF) creates most of this “merge tax”, which, according to Elastic’s research, can significantly increase query latency compared to single-index lookups.

Databases that integrate vector search with filtering, full-text search, and query execution in a single engine execute hybrid queries within a single atomic statement. The query optimizer can evaluate metadata filters, full-text conditions, and vector similarity at once. This lets the optimizer produce better execution plans and move less data.

In contrast, specialized vector silos fragment the query path. Applications route requests across multiple systems and merge results outside the database. This increases system complexity and introduces unpredictable latency under load.

Hybrid platforms such as Actian VectorAI DB address this problem by embedding vector search within the database engine. This design removes cross-system joins, simplifies operations, and reduces long-term architectural overhead.

Build Your Own Evaluation Framework

Stop asking which database won a GitHub leaderboard. Start asking which architecture survives your constraints. In 2026, these constraints center on data residency, scale, and team expertise.

The case for hybrid and on-premises

Data residency is no longer optional for global companies. With EU AI Act penalties reaching 35M Euros or 7% of global revenue, cloud-only vector databases represent a legal non-starter for regulated industries.

Sovereignty: 60% of financial firms outside the US plan to adopt sovereign/on-premises vector solutions by 2028.
Cost: As query volumes hit 100M/month, the “cloud tax” becomes visible. Self-hosting or using hybrid platforms like Actian can cut your infrastructure bill in half.
Maturity: If you already manage a relational database, your team possesses 90% of the required skills.

The 2026 architecture decision tree

Does the data require on-premises storage for compliance? → Prioritize Actian VectorAI DB or self-hosted PostgreSQL.
Does your query volume exceed 100M/month? → Avoid managed usage-based pricing; use self-hosted or reserved capacity.
Do you require complex metadata filtering? → An integrated relational/vector engine is non-negotiable.

How to Evaluate the Evaluators

To avoid letting vendor benchmarks mislead you, give the evaluation tool the same careful review you give the database. To spot a biased test, look past the headline QPS numbers and check the exact conditions that produced them.

Use the following evaluation rubric to review any benchmark report before it shapes your architectural decisions.

Evaluation metric	Red flag (Discard result)	Green flag (Trustworthy result)
Ingestion state	Queries run against a static, immutable index with zero background writes.	“Read-while-Write” testing, where queries run during continuous data ingestion.
Hardware parity	Vendor cloud “Optimized” vs. Competitor “Default” local/mismatched instances.	Verified identical CPU, RAM, and Disk I/O configurations across all tested systems.
Data selectivity	“High Selectivity” filters (99% of data removed) that hide join/scan inefficiencies.	“Low Selectivity” (10–20% filtered) tests that force the engine to handle large-scale index traversal.
Dimensionality	Testing on 128-dimension legacy datasets (SIFT/GIST).	Testing on 1,536 or 3,072-dimension vectors that match modern LLM outputs.
Latency metric	Focuses strictly on “Average Latency” or “Mean Response Time.”	Clearly publishes P95 and P99 tail latency under high concurrent load.

Pre-Commitment Checklist

Test with production-representative high-dimensional embeddings (3,072d+).
Measure P99 latency with 100+ concurrent users hitting diverse metadata filters.
Calculate 3-year TCO, including storage growth, egress, and re-indexing fees.
Confirm that your team can manage observability and backups for the new stack.

Final Thoughts

Real evaluation requires testing with your data, your patterns, and your scale. Load your production-representative data, run a week-long stability test under concurrent load, and measure P99 latency and the TCO.

If your workload requires compliance, hybrid deployment, or production-grade operational maturity that managed vector databases don’t offer, then Actian VectorAI DB early access is the right next step.

Join the Actian community on Discord to discuss vector architecture with engineers solving real production problems.