DEV Community

Alain Airom
Alain Airom

Posted on

The Secret to Efficient RAG: A Step-by-Step Guide to Chunking and Counting Your Vectors

#rag #tokens #vectorization #chunking

Debunking the vetor count estimation for documents

Introduction

In the often highly technical discussions of my day-to-day work, a frequent challenge is sizing a client’s Retrieval-Augmented Generation (RAG) solution — a task I often simplify by framing it as finding the perfect “T-shirt size.” To propose the ideal fit (Small, Medium, or Large) for their RAG pipeline, I must first gather key metrics: the total number of source documents, their average and maximum length (in tokens or characters), and the desired chunking strategy. These factors are critical because they directly dictate the total volume of data that must be vectorized and stored, which, in turn, impacts the choice of the vector database, the computational cost of embeddings, and the overall latency and performance of the RAG system. Without a solid estimation of the number of resultant vectors, our solution sizing will be guesswork. However, my audience is frequently diverse; while many of my interlocutors are highly skilled technical engineers eager to dive into the math, I equally often present to business-oriented decision-makers who need clarity over complexity. Regardless of the language — be it plain English or French — my explanation must distill the technical requirements into understandable concepts. I need to clearly articulate how and on what basis we estimate their resource usage, explaining that our sizing projection (the “T-shirt size”) is not arbitrary, but is a direct function of the estimated number of vectors derived from their document count and our chosen chunking strategy. This ensures that both the technical team and the business leadership understand the rationale behind the proposed solution and its associated costs and performance.

So to introduce the subject we must go through some definitions.

Vectorization

Vectorization is the process of converting textual data into numerical representations, called vectors or embeddings, which capture the semantic meaning of the text. Each vector is a sequence of numbers (e.g., [0.1, -0.5, 0.2, ...]) where its position in a high-dimensional space reflects its meaning, allowing similar texts to have vectors that are numerically "close" to each other. Chunking is the crucial prerequisite to vectorization in Retrieval-Augmented Generation (RAG) systems; it involves breaking down a large document into smaller, manageable segments (chunks) because most embedding models have a token limit and perform better on focused content. Thus, each chunk is then independently vectorized, resulting in a distinct vector for every chunk. This relationship means that the c*hunking strategy directly determines the number of vectors* created from a document, as each chunk ultimately becomes one vector that gets stored in a vector database for efficient retrieval based on semantic similarity.

Understand the Chunking Strategy

A document is rarely vectorized as a single entity. Instead, it’s broken down into smaller, manageable pieces called chunks. Each chunk will then become a single vector.

Key factors in chunking:

  • Chunk Size: This is the maximum number of characters or tokens allowed in a single chunk.
  • Calculation: A simple estimate is to divide the total number of characters or tokens in your document by the chosen chunk size.
  • Number of Chunks ≈ Total Characters (or Tokens) / Chunk Size

  • Overlap: Most chunking strategies use an overlap (e.g., 10–20% of the chunk size) between consecutive chunks to ensure context isn’t lost at the boundaries. Effect: Overlap slightly increases the total number of chunks (and thus, vectors) compared to the simple division above.
  • Splitters/Separators: The method uses to logically split the text (e.g., splitting by paragraphs, sentences, or a specific markdown heading). Effect: Logical splitting can result in chunks that are smaller than the maximum size, but you’ll have a number of chunks equal to the number of logical divisions found.

Practical Estimation Method

  • Count the Tokens/Characters: Use a token counter or a simple character counter to determine the document’s total size.
  • Apply the Chunking Formula: Use the formula: Number of Vectors = Total Tokens​ / Chunk Size−Overlap.

Understand the Embedding Model’s Role

The embedding model (e.g., text-embedding-ada-002, specific models from Cohere or Google) does not affect the number of vectors, but it determines the dimension of each vector.

  • Dimensionality: If a model generates a vector with 1536 floating-point numbers, the dimensionality of the vector is 1536. This is constant for a given model, regardless of the chunk’s content.

> In short: The number of vectors is determined by the number of chunks, not the embedding model itself.

The Definitive Way: Test and Count

The only way to get the exact number is to run the vectorization process once and count the resulting vectors.

Most vectorization frameworks (like LangChain, LlamaIndex, or custom scripts) perform these steps:

  • Load the document.
  • Split the document according to the configured chunking strategy (size, overlap, separators).
  • Embed each resulting chunk into a vector.
  • Store the vectors in a vector database.

The number of items stored in the vector database is your final count. For instance, in LangChain, if you use a document loader and a text splitter, the output will be a list of Document objects; the length of that list is the number of vectors that will be created.

Final Answer: to determine the number of vectors, first check your RAG’s chunking configuration and then divide the document’s total token/character count by the effective chunk size. The most accurate method is to execute the chunking step and count the resulting text segments.

This idea could be demonstrated as the following Python code ⤵️

import math

def calculate_vector_count(document_text: str, chunk_size: int, chunk_overlap: int) -> int:
    """
    Simulates character-based chunking to determine the number of vectors.

    Args:
        document_text: The full text of the document.
        chunk_size: The maximum number of characters per vector/chunk.
        chunk_overlap: The number of characters to repeat between adjacent chunks.

    Returns:
        The total number of chunks (vectors) created.
    """

    doc_length = len(document_text)

    # 1. Handle the simplest case: an empty document
    if doc_length == 0:
        return 0

    # 2. Handle the case where the document is smaller than the chunk size
    if doc_length <= chunk_size:
        return 1

    # 3. Calculate the effective new content added by each subsequent chunk
    # After the first chunk, each new chunk adds (chunk_size - chunk_overlap) characters.
    effective_chunk_size = chunk_size - chunk_overlap

    if effective_chunk_size <= 0:
        raise ValueError("Chunk overlap must be less than chunk size to make progress.")

    # 4. Determine how much content is left after the first chunk
    content_after_first_chunk = doc_length - chunk_size

    # 5. Calculate the number of ADDITIONAL chunks needed
    # We use math.ceil to account for any remaining characters
    additional_chunks = math.ceil(content_after_first_chunk / effective_chunk_size)

    # 6. The total count is 1 (the first chunk) + the additional chunks
    total_vectors = 1 + additional_chunks

    return total_vectors

# --- Setup for the Example ---

# A simple document for demonstration (1000 characters)
DOCUMENT = "A" * 1000  
# print(f"Document Length: {len(DOCUMENT)} characters")

# --- Scenario 1: Aggressive Chunking (Small Chunks, High Overlap) ---
CHUNK_SIZE_1 = 150
OVERLAP_1 = 30
vectors_1 = calculate_vector_count(DOCUMENT, CHUNK_SIZE_1, OVERLAP_1)

# --- Scenario 2: Conservative Chunking (Large Chunks, No Overlap) ---
CHUNK_SIZE_2 = 400
OVERLAP_2 = 0
vectors_2 = calculate_vector_count(DOCUMENT, CHUNK_SIZE_2, OVERLAP_2)

# --- Output ---
print(f"Document Length: {len(DOCUMENT)} characters")
print("-" * 50)
print("Scenario 1: Small Chunks, High Overlap (Aggressive)")
print(f"  Chunk Size: {CHUNK_SIZE_1} | Overlap: {OVERLAP_1}")
print(f"  Effective New Content per Chunk: {CHUNK_SIZE_1 - OVERLAP_1} characters")
print(f"  Total Number of Vectors: {vectors_1}")

print("-" * 50)
print("Scenario 2: Large Chunks, No Overlap (Conservative)")
print(f"  Chunk Size: {CHUNK_SIZE_2} | Overlap: {OVERLAP_2}")
print(f"  Effective New Content per Chunk: {CHUNK_SIZE_2 - OVERLAP_2} characters")
print(f"  Total Number of Vectors: {vectors_2}")
Enter fullscreen mode Exit fullscreen mode

The detailed article referenced at the end of this post provides an extensive exploration of chunking methodologies, illustrated through numerous practical examples. This critical resource formed the cornerstone of my analysis and informed the insights shared within this article.

End-to-End Sample Application

A sample application which caculates all described above.

  • As usual prepare your environment ⬇️ 
python3 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install langchain pypdf
pip install langchain-community
Enter fullscreen mode Exit fullscreen mode
  • And below is the code which uses a sample document as input. LangChain is used in this sample application because it provides a structured, modular, and developer-friendly framework for building applications powered by large language models (LLMs) and (almost) universally known by everyone. 
import os
from glob import glob
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# --- Configuration ---
INPUT_DIR = "./input"
OUTPUT_DIR = "./output"
RESULTS_FILE = "chunk_counts.txt"

# LangChain Chunker Configuration
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

def create_directory_if_not_exists(directory_path: str):
    """Creates a directory if it doesn't already exist."""
    os.makedirs(directory_path, exist_ok=True)
    print(f"Ensuring directory exists: {directory_path}")

def load_documents(file_path: str):
    """Loads a document based on its file extension."""

    # Get the file extension 
    extension = os.path.splitext(file_path)[1].lower()

    if extension == ".pdf":
        print(f"  -> Loading PDF: {os.path.basename(file_path)}")
        loader = PyPDFLoader(file_path)
    elif extension in [".txt", ".md"]:
        print(f"  -> Loading Text: {os.path.basename(file_path)}")
        loader = TextLoader(file_path)
    else:
        print(f"  -> Skipping unsupported file type: {os.path.basename(file_path)}")
        return None

    return loader.load()

def main():
    """
    Main function to process documents, chunk them, and save the counts.
    """

    # 1. Setup Input and Output Directories
    create_directory_if_not_exists(OUTPUT_DIR)

    # Find all supported files in the input directory
    search_pattern = os.path.join(INPUT_DIR, "*.*")
    all_files = glob(search_pattern)

    if not all_files:
        print(f"\nNo documents found in '{INPUT_DIR}'. Please add files and run again.")
        return

    # 2. Initialize the LangChain Text Splitter (The "Chunker")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        length_function=len,
        separators=["\n\n", "\n", " ", ""] # Try to split logically first
    )
    print("\n--- LangChain Chunker Initialized ---")
    print(f"Chunk Size: {CHUNK_SIZE} | Overlap: {CHUNK_OVERLAP}\n")

    # 3. Process Documents
    total_chunks_processed = 0
    results = []

    for file_path in all_files:
        # Load the document content
        documents = load_documents(file_path)

        if documents is None or not documents:
            continue

        # 4. Chunk the documents
        # The split_documents method handles breaking the text into smaller Document objects
        chunks = text_splitter.split_documents(documents)

        num_chunks = len(chunks)
        total_chunks_processed += num_chunks

        # 5. Record the result
        file_name = os.path.basename(file_path)
        result_line = f"{file_name}: {num_chunks} vectors (chunks)"
        results.append(result_line)
        print(f"  -> Result: {num_chunks} chunks created.")

    # 6. Write Output to File
    output_path = os.path.join(OUTPUT_DIR, RESULTS_FILE)

    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(f"--- Document Chunking Results ---\n")
        f.write(f"Chunk Size: {CHUNK_SIZE} | Overlap: {CHUNK_OVERLAP}\n\n")
        for line in results:
            f.write(line + "\n")
        f.write(f"\nTOTAL VECTORS/CHUNKS PROCESSED: {total_chunks_processed}\n")

    print(f"\n--- Processing Complete ---")
    print(f"Total Vectors/Chunks created: {total_chunks_processed}")
    print(f"Results saved to: {output_path}")

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode
  • The output of the application is shown below 👇 
--- Document Chunking Results ---
Chunk Size: 1000 | Overlap: 200

enterprise-management.pdf: 2841 vectors (chunks)

TOTAL VECTORS/CHUNKS PROCESSED: 2841
Enter fullscreen mode Exit fullscreen mode

Et voilà ⛳

Bonus 🔷

After writing the first application, I wanted to enhance it, so it could estimate the number of sentences and also the length for a given document, to achieve a result such as the one provided hereafter.

--- Document Chunking & Estimation Results ---
Chunk Size: 1000 | Overlap: 200

--- Overall Document Statistics (before chunking) ---
  Total Raw Characters Across All Processed Files: 2067187
  Total Raw Tokens (tiktoken est.) Across All Processed Files: 414873
  Total Raw Estimated Sentences Across All Processed Files: 14180
  Average Chars per Processed Document: 2067187.00
  Average Tokens per Processed Document: 414873.00
  Average Sentences per Processed Document: 14180.00

--- Detailed File Results ---
  enterprise-management.pdf:
    - Original Text Length (Chars): 2067187
    - Original Text Length (Tokens - tiktoken est.): 414873
    - Original Text (Estimated Sentences): 14180
    - Chunks Created (Vectors): 2841

--- Summary ---
TOTAL VECTORS/CHUNKS PROCESSED ACROSS ALL DOCUMENTS: 2841
Enter fullscreen mode Exit fullscreen mode

➡️ So here goes the version 2 of my application.

import os
from glob import glob
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken
import nltk
import re # For some basic text cleaning

# --- NLTK Punkt Downloader ---
# Ensures all necessary NLTK tokenizer data is available.
nltk_resources = ['punkt', 'punkt_tab'] # List of NLTK resources to check/download

print("--- Checking NLTK Data ---")
for resource in nltk_resources:
    try:
        # Attempt to find the resource first
        nltk.data.find(f'tokenizers/{resource}')
        print(f"NLTK '{resource}' tokenizer data already available.")
    except LookupError: # Catch the specific LookupError if resource is not found
        print(f"NLTK '{resource}' tokenizer data not found. Downloading...")
        nltk.download(resource)
        print(f"NLTK '{resource}' data downloaded.")
print("--- NLTK Data Check Complete ---\n")
# End of NLTK download block

# --- Configuration ---
INPUT_DIR = "./input"
OUTPUT_DIR = "./output"
RESULTS_FILE = "chunk_counts_enhanced.txt"

# LangChain Chunker Configuration
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200

# --- Tiktoken Token Counting ---
# Using the encoder for 'cl100k_base' which is used by models like gpt-4, gpt-3.5-turbo
TOKENIZER = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    """Counts tokens using a tiktoken encoder."""
    return len(TOKENIZER.encode(text))

def count_sentences(text: str) -> int:
    """Counts sentences using NLTK's punkt tokenizer."""
    # Ensure text is not empty or just whitespace to avoid issues with nltk.sent_tokenize
    if not text or not text.strip():
        return 0
    sentences = nltk.sent_tokenize(text)
    return len(sentences)

def create_directory_if_not_exists(directory_path: str):
    """Creates a directory if it doesn't already exist."""
    os.makedirs(directory_path, exist_ok=True)
    print(f"Ensuring directory exists: {directory_path}")

def load_documents(file_path: str):
    """Loads a document based on its file extension."""
    extension = os.path.splitext(file_path)[1].lower()

    # Langchain loaders return a list of Document objects
    # For single file, we expect one Document object primarily, but PDFs can have multiple pages
    loaded_docs = []

    if extension == ".pdf":
        print(f"  -> Loading PDF: {os.path.basename(file_path)}")
        loader = PyPDFLoader(file_path)
        loaded_docs = loader.load()
    elif extension in [".txt", ".md", ".json", ".csv"]: # Added more common text types
        print(f"  -> Loading Text: {os.path.basename(file_path)}")
        # Specify encoding for robustness. 'utf-8' is a good default.
        loader = TextLoader(file_path, encoding='utf-8') 
        loaded_docs = loader.load()
    else:
        print(f"  -> Skipping unsupported file type: {os.path.basename(file_path)}")
        return None

    return loaded_docs

def main():

    # 1. Setup Input and Output Directories
    create_directory_if_not_exists(OUTPUT_DIR)

    # Find all supported files in the input directory
    search_pattern = os.path.join(INPUT_DIR, "*.*")
    all_files = glob(search_pattern)

    if not all_files:
        print(f"\nNo documents found in '{INPUT_DIR}'. Please add files and run again.")
        return

    # 2. Initialize the LangChain Text Splitter (The "Chunker")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=CHUNK_SIZE,
        chunk_overlap=CHUNK_OVERLAP,
        length_function=len, # Use character length for splitting
        separators=["\n\n", "\n", " ", ""] # Prioritize logical breaks
    )
    print("--- LangChain Chunker Initialized ---")
    print(f"Chunk Size: {CHUNK_SIZE} | Overlap: {CHUNK_OVERLAP}\n")

    # 3. Process Documents
    total_chunks_processed = 0
    total_raw_chars = 0
    total_raw_tokens = 0
    total_raw_sentences = 0
    num_processed_files = 0 # To accurately calculate averages for files actually processed

    results = []

    for file_path in all_files:
        file_name = os.path.basename(file_path)
        print(f"\nProcessing file: {file_name}")

        # Load the document content - this can return multiple Document objects for e.g. multi-page PDFs
        documents = load_documents(file_path)

        if documents is None or not documents:
            continue

        num_processed_files += 1

        # Concatenate text from all parts of the document (e.g., all pages of a PDF)
        # Each 'doc' in 'documents' is a LangChain Document object with 'page_content'
        full_text = " ".join([doc.page_content for doc in documents])

        # --- Basic Text Cleaning (Optional but Recommended for better estimates) ---
        # Remove excessive whitespace, replace multiple newlines with single spaces
        # This helps in more accurate token and sentence counts by removing irrelevant noise
        full_text = re.sub(r'\s+', ' ', full_text).strip()

        # --- Estimate Raw Document Metrics ---
        char_count = len(full_text)
        token_count = count_tokens(full_text)
        sentence_count = count_sentences(full_text)

        total_raw_chars += char_count
        total_raw_tokens += token_count
        total_raw_sentences += sentence_count

        # 4. Chunk the documents
        # The split_documents method takes a list of Document objects and returns a list of smaller Document objects
        chunks = text_splitter.split_documents(documents)

        num_chunks = len(chunks)
        total_chunks_processed += num_chunks

        # 5. Record the result for this file
        result_line = (
            f"  {file_name}:\n"
            f"    - Original Text Length (Chars): {char_count}\n"
            f"    - Original Text Length (Tokens - tiktoken est.): {token_count}\n"
            f"    - Original Text (Estimated Sentences): {sentence_count}\n"
            f"    - Chunks Created (Vectors): {num_chunks}"
        )
        results.append(result_line)
        print(f"  -> Chunks created for {file_name}: {num_chunks}")

    # 6. Write Output to File
    output_path = os.path.join(OUTPUT_DIR, RESULTS_FILE)

    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(f"--- Document Chunking & Estimation Results ---\n")
        f.write(f"Chunk Size: {CHUNK_SIZE} | Overlap: {CHUNK_OVERLAP}\n\n")

        f.write(f"--- Overall Document Statistics (before chunking) ---\n")
        f.write(f"  Total Raw Characters Across All Processed Files: {total_raw_chars}\n")
        f.write(f"  Total Raw Tokens (tiktoken est.) Across All Processed Files: {total_raw_tokens}\n")
        f.write(f"  Total Raw Estimated Sentences Across All Processed Files: {total_raw_sentences}\n")

        # Calculate averages only if files were actually processed
        if num_processed_files > 0:
            f.write(f"  Average Chars per Processed Document: {total_raw_chars / num_processed_files:.2f}\n")
            f.write(f"  Average Tokens per Processed Document: {total_raw_tokens / num_processed_files:.2f}\n")
            f.write(f"  Average Sentences per Processed Document: {total_raw_sentences / num_processed_files:.2f}\n")
        else:
            f.write("  No documents were successfully processed to calculate averages.\n")

        f.write(f"\n--- Detailed File Results ---\n")
        for line in results:
            f.write(line + "\n\n")
        f.write(f"--- Summary ---\n")
        f.write(f"TOTAL VECTORS/CHUNKS PROCESSED ACROSS ALL DOCUMENTS: {total_chunks_processed}\n")

    print(f"\n--- Processing Complete ---")
    print(f"Total Vectors/Chunks created across all documents: {total_chunks_processed}")
    print(f"Results saved to: {output_path}")

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Et re-voilà 😅

Conclusion

In conclusion, this exercise tries tp meticulously guide through the practical implementation of a core RAG pipeline component: document preparation and vector count estimation. We’ve seen how a Python application can systematically scour an input directory for documents, leveraging LangChain to chunk content according to specified parameters like size and overlap. The resulting calculation of the number of vectors provides invaluable insight for sizing a RAG solution, translating directly into estimations for vector database storage, embedding costs, and overall system performance — a critical bridge between technical specifics and business requirements.

Links

Top comments (0)