Build a Local RAG Pipeline on Declassified UFO Files with VectorAI DB and Ollama

Production document collections rarely look like benchmark datasets. Some PDFs contain searchable text. Others are scanned pages that need OCR. Metadata varies across sources, and retrieval pipelines that work on clean examples often struggle once those differences appear.

In this tutorial, you'll build a fully local RAG pipeline around declassified U.S. government UFO files from the PURSUE archive. Using Ollama and VectorAI DB, you'll extract text, handle scanned documents, attach metadata, and generate cited answers without relying on external APIs.

The UFO files are simply the dataset. The same pipeline applies to internal documentation, legal discovery archives, and research repositories where documents come from different sources and follow different formats.

By the end, you'll have a single-machine RAG system that ingests PDFs, stores embeddings with metadata, filters results by agency or decade, and generates responses grounded in the source documents.

What You Are Building

The system has three layers, all running on a single machine.

Ingestion: PDFs from the PURSUE archive pass through an ingestion pipeline that handles both machine-readable and scanned documents. PyMuPDF extracts text where possible, while pytesseract processes image-based pages. The text is chunked, embedded locally with Ollama's nomic-embed-text, and stored in VectorAI DB with metadata: agency, decade, source filename, and source URL.

Retrieval: Queries are embedded with the same model used during ingestion. VectorAI DB returns the nearest chunks and can narrow the search using metadata filters before ranking by similarity.

Generation: Retrieved chunks and the user's question are sent to llama3.2, which generates an answer grounded in the retrieved documents and includes references back to the original files.

Everything runs locally on a single machine. A GPU speeds up embedding and generation, but the pipeline works on CPU-only hardware.

Figure 1: System architecture diagram. Three layers stacked vertically inside a single machine boundary.

Set Up the Environment

Get VectorAI DB and Ollama running, then install the Python dependencies.

Create a docker-compose.yml file in your project folder:

services:
  vectorai:
    image: actian/vectorai:latest
    container_name: vectorai-ufo
    ports:
      - "6573:6573"
      - "6574:6574"
      - "6575:6575"
    volumes:
      - ./local_data:/var/lib/actian-vectorai
    environment:
      - ACTIAN_VECTORAI_ACCEPT_EULA=YES
    restart: unless-stopped

Start it:

docker-compose up -d

The image exposes REST on port 6573, gRPC on port 6574, and a local web UI on 6575. Verify the database is running:

docker logs vectorai-ufo

You should see Ready to accept connections... near the end of the log, along with your license tier and current vector count:

Figure 2: Docker logs

Install Ollama from ollama.com/download. Pull both models:

ollama pull nomic-embed-text
ollama pull llama3.2

Install Python dependencies:

pip install actian-vectorai-client pymupdf pytesseract ollama

Download and Inspect the Document Collection

The PURSUE Release 01 is available directly at war.gov/UFO/. The release contains 162 files: 120 PDFs, 28 videos, and 14 high-resolution images. This tutorial ingests the PDFs only. A second release on May 23, 2026 added 64 more files.

For faster onboarding, the community project DenisSergeevitch/UFO-USA has all 120 PDFs converted to Markdown via Gemini OCR, totaling 4,185 pages. Clone it to skip the OCR step during development:

git clone https://github.com/DenisSergeevitch/UFO-USA.git mirror

This tutorial reads from the Markdown archive but also shows the PyMuPDF and pytesseract code so you understand how to handle raw PDFs on your own document collection.

The PURSUE collection mixes two PDF types. State Department memos and NARA records have extractable text layers and parse cleanly with PyMuPDF. The major FBI case file (62-HQ-83894, covering UFO investigations from 1947 to 1968, totaling over 1,100 pages across multiple sections) consists of scanned images and requires OCR. Your ingestion pipeline branches on PDF type because the same code has to handle both.

Figure 3: Side-by-side comparison of a machine-readable State Department PDF page and a scanned FBI case file page, showing why the OCR branch exists.

Build the Ingestion Pipeline

Walk through ingestion in six steps. Every code block runs.

Step 1 Detect the PDF type: Try to extract text with PyMuPDF first. If the page yields fewer than 50 characters, treat it as scanned and route to OCR.

import fitz  # PyMuPDF
import pytesseract
from PIL import Image
import io
def extract_pdf_text(path: str) -> str:
    """Extract text from a PDF, routing scanned pages through OCR."""
    doc = fitz.open(path)
    all_text = []

    for page in doc:
        text = page.get_text().strip()

        # Fewer than 50 chars on a non-empty page usually means scanned.
        if len(text) < 50:
            pix = page.get_pixmap(dpi=200)
            img = Image.open(io.BytesIO(pix.tobytes("png")))
            text = pytesseract.image_to_string(img)

        all_text.append(text)

    doc.close()
    return "\n\n".join(all_text)

Step 2. Chunk the text: Use 700-word chunks with 70-word overlap. This stays comfortably within the embedding model's context window while preserving enough surrounding text to keep related ideas together during retrieval.

def chunk_text(text: str, size: int = 700, overlap: int = 70):
    """Split text into overlapping word chunks."""
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = " ".join(words[i:i + size])
        if chunk.strip():
            chunks.append(chunk)
        i += size - overlap
    return chunks

Step 3. Embed each chunk: Call Ollama's embeddings endpoint with nomic-embed-text. The model returns a 768-dimensional vector for each chunk.

import ollama
def embed(text: str):
    """Return a 768-dim embedding for the input text."""
    resp = ollama.embeddings(model="nomic-embed-text", prompt=text)
    return resp["embedding"]

Step 4. Create the VectorAI DB collection: The schema holds the 768-dimensional vector plus metadata fields for filtering: agency, decade, document type, source filename, and the direct war.gov URL for the source PDF.

from actian_vectorai import VectorAIClient, VectorParams, Distance

COLLECTION = "pursue_demo"

with VectorAIClient("localhost:6574") as client:
    client.collections.get_or_create(
        name=COLLECTION,
        vectors_config=VectorParams(size=768, distance=Distance.Cosine),
    )

API method names and configuration keys in this tutorial are verified against docs.vectoraidb.actian Check the reference before deploying to production, as signatures may change between SDK versions.

Step 5. Classify each document by agency and decade: Derive these from the filename and store them in the payload so retrieval can filter on them later.

import re
def classify_agency(filename: str) -> str:
    name = filename.lower()
    if "fbi" in name or "62-hq" in name:
        return "FBI"
    if "nasa" in name:
        return "NASA"
    if "state" in name or "342" in name:
        return "State"
    if "odni" in name:
        return "ODNI"
    return "Unknown"

def derive_decade(filename: str) -> int:
    years = re.findall(r"(19[4-9][0-9]|20[0-2][0-9])", filename)
    if not years:
        return 0
     return (int(years[0]) // 10) * 10

Step 6. Insert chunks with metadata: Keep the client connection open for the full ingestion run. Opening and closing a connection per document adds overhead and can trigger server-side keepalive limits on long embedding passes.

import os
import uuid
from pathlib import Path
from actian_vectorai import PointStruct

def ingest_document(client, text: str, doc_name: str, source_url: str = ""):
    chunks = chunk_text(text)
    agency = classify_agency(doc_name)
    decade = derive_decade(doc_name)

    points = []
    for chunk in chunks:
        vector = embed(chunk)
        points.append(PointStruct(
            id=uuid.uuid4().int >> 64,
            vector=vector,
            payload={
                "content": chunk,
                "agency": agency,
                "decade": decade,
                "document_type": "scanned" if agency == "FBI" else "machine_readable",
                "source_doc": doc_name,
                "source_url": source_url,
            },
        ))

    if points:
        client.points.upsert(COLLECTION, points=points)
    return len(points)


# Run ingestion with a single open connection
with VectorAIClient("localhost:6574") as client:
    total = 0
    converted = Path("mirror/converted")
    for doc_name in sorted(os.listdir(converted)):
        doc_dir = converted / doc_name
        if not doc_dir.is_dir():
            continue
        full_text = ""
        for page_file in sorted(doc_dir.glob("page-*.md")):
            full_text += page_file.read_text(encoding="utf-8") + "\n\n"
        total += ingest_document(client, full_text, doc_name)
    client.vde.flush(COLLECTION)
    print(f"Done. {total} chunks inserted.")

Figure 4: ingestion process

client.vde is the Vector Data Engine handle for low-level storage operations. Calling flush() after the ingestion loop ensures all writes are committed before the session ends, preventing data loss if the process stops immediately after ingestion.

Run Your First Query

Retrieval works by embedding the query with the same model used at ingestion time, then finding the closest vectors in the collection.

def retrieve(query: str, k: int = 5):
    query_vector = embed(query)
    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
        ) or []


results = retrieve("flying disc sightings reported to government agencies")

for r in results:
    p = r.payload
    print(f"score={r.score:.3f}  agency={p['agency']:<8}  doc={p['source_doc'][:50]}")
    print(f"  {p['content'][:200]}...\n")

Running this returns:

Figure 5: Retreival process
The query retrieves both State Department records and FBI files, even though the wording differs across documents. Retrieval is based on semantic similarity rather than exact keyword matches, allowing related material from different agencies to surface together.

Add Metadata Filters

Metadata filtering runs before similarity ranking, so only documents matching the filter are scored. The same query returns different document sets depending on which agency you scope it to.

from actian_vectorai import Field, FilterBuilder

def retrieve_filtered(query: str, agency: str = None, decade: int = None, k: int = 5):
    query_vector = embed(query)
    builder = FilterBuilder()
    if agency:
        builder = builder.must(Field("agency").eq(agency))
    if decade:
        builder = builder.must(Field("decade").eq(decade))
    filter_obj = builder.build() if (agency or decade) else None

    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
            filter=filter_obj,
        ) or []

Run the same query three ways:
# Unfiltered: results across all agencies
all_results = retrieve_filtered("flying disc sighting")

# FBI documents only
fbi_results = retrieve_filtered("flying disc sighting", agency="FBI")

# State Department documents only
state_results = retrieve_filtered("flying disc sighting", agency="State")

On a company knowledge base you'd filter by department or document type. On a legal discovery archive you'd filter by custodian or date range. The pattern is the same.

Figure 6:Query Results

Wire In the Local LLM

Connect retrieval to generation. The LLM receives the user's question and the retrieved chunks, and produces a grounded answer with citations.

def answer(query: str, k: int = 5, agency: str = None):
    results = retrieve_filtered(query, agency=agency, k=k)

    context_blocks = []
    for i, r in enumerate(results, 1):
        p = r.payload
        context_blocks.append(
            f"[Source {i}] Agency: {p['agency']} | Document: {p['source_doc']}\n"
            f"{p['content']}\n"
        )
    context = "\n---\n".join(context_blocks)

    prompt = f"""Answer the question using only the sources below. Cite each claim \
by source number in square brackets, like [Source 2]. If the sources do not answer \
the question, say so.

SOURCES:
{context}

QUESTION: {query}

ANSWER:"""

    response = ollama.generate(model="llama3.2", prompt=prompt, stream=True)
    for chunk in response:
        print(chunk["response"], end="", flush=True)
    print()

    print("\nSources cited:")
    for i, r in enumerate(results, 1):
        p = r.payload
        print(f"  [{i}] {p['agency']:<8} {p['source_doc']}")
        if p.get("source_url"):
            print(f"         {p['source_url']}")

Running the pipeline end-to-end against the 1949 State Department documents:

Query: What did witnesses report about flying discs in 1949?

Figure 7:LLM reuslts

Each statement in the answer comes from retrieved documents. The source links make it possible to verify the underlying material directly. The answer cites specifics: delta-shaped, no banking on the 180-degree turn, surface color between light gray and dirty white. These details come from the 1949 USAF intelligence memo, not from the model's training data. The whole response ran on a laptop with no API key required. Generation time depends on your CPU and the response length. Expect anywhere from 30 seconds to a few minutes on CPU-only hardware.

Three Substitution Paths

The architecture is not specific to the PURSUE files.

Swap the document collection: Replace the PURSUE files with company documentation, legal records, clinical trial archives, or any other PDF collection. The ingestion pipeline, VectorAI DB schema, and retrieval logic stay the same. Update classify_agency and derive_decade to extract whatever metadata matters for your collection: department, project ID, jurisdiction, author. Anything stored in the payload at ingestion time becomes a filter at retrieval time.

Scale to the full document collection: This tutorial demonstrates the architecture on a focused subset. The GitHub repository includes a full ingestion script that processes all 120 PDFs across the five agencies in PURSUE Release 01. On CPU, the full run takes several hours. On a GPU-enabled machine, the embedding pass takes under 30 minutes. The Community Edition license supports up to 5,000 vectors, which covers the full Release 01 collection at these chunk settings. Release 02 (May 23, 2026) added 64 more files. A scheduled job that runs the ingestion pipeline on new downloads keeps the collection current.

Deploy on any infrastructure: The Docker Compose file runs identically on a laptop, a server, an edge device, or an air-gapped environment. VectorAI DB is the same across all of them. The hardware changes, not the architecture or the application code.

Wrapping Up

You built a fully local RAG pipeline that handles mixed-format documents, stores embeddings with metadata, retrieves relevant chunks, and generates cited answers using Ollama and VectorAI DB.

The UFO archive is only one example. The same approach applies to company documentation, legal records, research repositories, and other collections where documents arrive from different sources and follow different formats.

In practice, messy document collections are often the hard part of RAG systems. Building a pipeline that can handle those differences matters more than the dataset itself.

The complete code, Docker Compose configuration, and requirements are available in the GitHub repository. The accompanying README includes instructions for processing the full PURSUE collection and adapting the pipeline to other document libraries.

For deeper VectorAI DB implementation details, see the VectorAI DB Python SDK reference.