DEV Community

Cover image for Local-First RAG Pipeline in Pure Python
abdullah bin aqeel
abdullah bin aqeel

Posted on

Local-First RAG Pipeline in Pure Python

#opensource #rag #machinelearning #ai

Stop Wasting Cloud Budget: Ingest & Chunk Unstructured Local Data in Milliseconds

Let's be honest: building Retrieval-Augmented Generation (RAG) pipelines, context-aware LLM agents, or local semantic search engines is incredibly exciting. What isn't exciting is wasting days writing boilerplate ingestion logic or pulling in bloated, multi-gigabyte frameworks just to split a folder of local Markdown and text files.

When did parsing a local directory and breaking text into clean chunks become so heavy? Monolithic AI orchestrators introduce massive dependency chains, require internet connectivity to calculate basic token metrics, and frequently mangle structural context by splitting text arbitrarily mid-word or mid-sentence.

To solve this exact engineering friction point, I built NexusFlow—a high-performance, zero-configuration local data pipeline engine designed to turn messy local text streams into optimized, context-preserving semantic chunks entirely offline.

Interactive Repository

Check out the full source code, architecture logs, and contribution guidelines natively on GitHub:

GitHub logo Abdullahbinaqeel / RAGMill

RAGMill

PyPI CI License: MIT Python

A lightweight, zero-config local pipeline engine for AI data ingestion, semantic chunking, embeddings, and vector search.

Install

pip install ragmill[all]   # includes PDF + DOCX + embeddings support
# or
pip install ragmill        # core only (txt/md), zero dependencies
Enter fullscreen mode Exit fullscreen mode

Developing locally instead? Clone the repo and use an editable install:

pip install -e ".[dev]"
pytest tests/ -v
Enter fullscreen mode Exit fullscreen mode

Usage

Ingest + chunk

from ragmill import RAGEngine

engine = RAGEngine(chunk_size=500, overlap=50)
chunks = engine.execute_pipeline("./my_documents")

for chunk in chunks:
    print(chunk["metadata"]["filename"], chunk["content"][:80])
Enter fullscreen mode Exit fullscreen mode

Supports .txt, .md, .log, .rst, .pdf, and .docx out of the box.

Embed + search locally

Requires the embeddings extra (pip install -e ".[embeddings]"). The model (a quantized MiniLM ONNX export, ~23MB) downloads once to ~/.cache/ragmill/models…

View on GitHub

(If you find this project useful for your local workflows, feel free to drop a STAR to support open-source development!)

The Solution: High-Performance Local-First Architecture

RAGMill breaks data preparation into three highly efficient, completely decoupled stages:

  1. The Ingestion Engine: An asynchronous, zero-copy structural directory crawler that streams raw text while keeping your system's memory consumption flat.
  2. The Semantic Chunking Router: Rejects naive character-based thresholds. It uses recursive boundary analysis to split text at natural structural breaks (like double-newlines, markdown headers, and sentence punctuation).
  3. Context Preservation: Dynamically manages sliding-window overlaps to carry vital context across boundaries so your downstream embedding vectors retain optimal meaning.

Quick Start (Under 30 Seconds)

RAGMill is written with strict type-safety, ensuring instant autocomplete support inside modern code editors like Cursor or VS Code.

Install it cleanly from PyPI:

pip install ragmill[all]
Enter fullscreen mode Exit fullscreen mode

The Architectural Deep Dive

Why Custom Regex Over Heavy Tokenizer Dependencies?

Standard tokenizers (like tiktoken or HuggingFace transformers) introduce heavy binaries and active model weight requirements into your environment just to calculate safe text splitting boundaries.

RAGMill optimizes this process using a custom deterministic regex routine. By scoring structural text markers recursively:

  • It keeps logically connected paragraphs completely whole.
  • If a paragraph exceeds the maximum chunk size, it effortlessly steps down to sentence-level token tracking.
  • It safely falls back to safe string slices only as an absolute last resort, keeping execution speed down to a fraction of a millisecond.

The output payload is a highly structured, model-ready JSON-like dictionary containing granular file manifests, index tracking, and character lengths—ready to dump straight into local caches, NumPy arrays, or vector layers like SQLite or Cloud Firestore.

Roadmap & Contributing

RAGMill is built with a lightweight footprint to prevent dependency conflicts across your production applications. Moving forward, the roadmap includes:

  • [ ] Native multi-threaded PDF and unformatted JSON parsing layers.
  • [ ] An ultra-lightweight ONNX runtime vector embedding execution module.
  • [ ] Out-of-the-box pipeline export adapters for common vector databases.

As an open-source project, contributions are incredibly welcome! If you want to add support for extra file abstractions or optimize the regex array, check out our GitHub repository, take a look at the good-first-issue tags, and submit a PR.

How are you optimizing your text ingestion pipelines for RAG systems today? Let's discuss in the comments below!

GitHub: https://github.com/Abdullahbinaqeel/RAGMill
PyPI: https://pypi.org/project/ragmill/
Medium: https://medium.com/@abdulbinaqeel/local-first-rag-pipeline-in-pure-python-1f8fbc79b5de?sharedUserId=abdulbinaqeel

Top comments (0)