Mustafa ERBAY

Posted on Aug 4 • Originally published at mustafaerbay.com.tr

RAG Chunking Test: The Right Chunk Size for Turkish Documents

#rag #guide #software

Providing accurate and relevant information from external sources to large language models (LLMs) in Retrieval-Augmented Generation (RAG) applications is critical for the system's success. One of the fundamental steps in this process is breaking down text documents into meaningful segments (chunks), and finding the correct chunk size, especially in agglutinative languages like Turkish, is a complex engineering problem. Incorrect chunking strategies can lead to LLMs generating irrelevant responses or overlooking critical information.

In this post, we will delve into RAG chunking testing processes for Turkish documents, explore different strategies, and discuss approaches to determining the optimal chunk size. Our goal is to provide practical solutions that enable LLMs to generate more accurate and contextually appropriate answers from Turkish texts.

What is RAG Chunking and Why is it Important?

RAG (Retrieval-Augmented Generation) is an architecture that allows LLMs to go beyond their training data and retrieve relevant information from an external knowledge base, enabling them to produce more up-to-date, accurate, and contextually appropriate responses. This approach reduces the tendency of LLMs to "hallucinate" while enhancing their expertise in specific domains. At the core of RAG is the ability to quickly find the most relevant information chunks for a user's query and present them to the LLM.

Chunking is one of the first and most critical steps in the RAG process; it is the operation of dividing documents into small, meaningful segments that can be processed by embedding models and fit within the LLM's context window. If chunks are too small, context can be lost, and semantic integrity can be compromised. On the other hand, if chunks are too large, the retriever can introduce noise by including irrelevant information, making it difficult for the LLM to find the relevant part.

ℹ️ Context Window and Chunking

LLMs have a limited context window. The amount of information that can fit into this window directly affects the model's performance. Chunking is indispensable for utilizing this limited window most efficiently.

We can simply visualize the RAG flow as follows:

The Impact of the Turkish Language on Chunking

Turkish is linguistically an agglutinative language; meaning, new words are formed or meanings are changed by adding various suffixes to root words. For example, the word "ev" (house) can express a complex meaning in a single word by taking multiple suffixes like "ev-ler-imiz-den" (from our houses). This structure significantly differs from analytical languages like English.

This linguistic characteristic directly affects chunking strategies. While word or token-based chunking often yields good results for English texts, this can be problematic in Turkish. A "token" in English usually corresponds to a word, whereas in Turkish, highly agglutinated words might be counted as a single token, or some tokenizers might break these words into multiple sub-parts. This necessitates caution when determining chunk_size and chunk_overlap parameters. The rich morphological structure of Turkish makes it difficult to define semantic boundaries and creates situations where standard delimiters (comma, period) do not always form a meaningful contextual segment.

Chunking Strategies and Parameters

There are various chunking strategies used in RAG systems. Each has its own advantages and disadvantages, and their performance on Turkish texts can vary. Choosing the right strategy directly impacts the quality of retrieval.

Fixed-Size Chunking

This is the simplest and most common chunking method. Documents are divided into fixed-size chunks with a specific chunk_size and chunk_overlap. chunk_overlap helps maintain context between chunks.

from langchain.text_splitter import CharacterTextSplitter

text_splitter = CharacterTextSplitter(
    separator="\n\n", # Paragraph separator
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    is_separator_regex=False,
)

# Example usage
long_text = "This is a long Turkish text. The first paragraph ends here.\n\nThe second paragraph begins and contains information. This paragraph also ends here.\n\nThe third and final paragraph. It contains an important detail."
chunks = text_splitter.split_text(long_text)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1} ({len(chunk)} characters):\n{chunk}\n---")

The advantage of this method is its simplicity and predictability. Its disadvantage is that it can cut sentences or paragraphs in half, ignoring semantic boundaries. In Turkish texts, using character count as chunk_size can yield more stable results than token count in some cases due to the length of words and sentence structures.

Recursive Character Text Splitter

This splitter uses a list of separators to divide text and tries these separators in a hierarchical order. For example, it first tries double newlines (\n\n), then single newlines (\n), then spaces (), and finally, if no separator is found, it splits by character. This way, it performs smarter splitting by preserving semantic integrity as much as possible.

from langchain.text_splitter import RecursiveCharacterTextSplitter

recursive_splitter = RecursiveCharacterTextSplitter(
    separators=["\n\n", "\n", " ", ""], # Separator hierarchy
    chunk_size=500,
    chunk_overlap=100,
    length_function=len,
    is_separator_regex=False,
)

# Example usage
long_text_recursive = "This is a long Turkish text. The first paragraph ends here.\n\nThe second paragraph begins and contains information. This paragraph also ends here.\n\nThe third and final paragraph. It contains an important detail."
chunks_recursive = recursive_splitter.split_text(long_text_recursive)
for i, chunk in enumerate(chunks_recursive):
    print(f"Chunk {i+1} ({len(chunk)} characters):\n{chunk}\n---")

RecursiveCharacterTextSplitter is a recommended method by LangChain for general texts and offers a better starting point than Fixed-Size Chunking for complex languages like Turkish, as it has a higher potential to capture semantic boundaries. However, the optimal chunk_size and chunk_overlap values still need to be found through trial and error.

Semantic Chunking

Semantic chunking focuses on the semantic content of segments rather than directly dividing text by character or token count. This method creates embeddings of sentences or paragraphs in the text and groups semantically similar ones to form chunks. This allows for higher quality and more contextually coherent chunks.

💡 Semantic Chunking and Turkish

In Turkish texts, especially long and technical documents, semantic chunking can be more effective in overcoming the challenges arising from the language's morphological structure. However, this approach requires more computational cost, as embeddings need to be generated for each small segment.

This method typically works with a threshold value; a new chunk is initiated when the embedding similarity drops below this threshold. Tools like SemanticChunker in libraries like LangChain allow for this type of approach. However, for our test-oriented approach in this article, we will primarily focus on Fixed-Size and RecursiveCharacterTextSplitter.

Enriching with Metadata

Adding metadata such as source information (file name, URL), page number, section title, and date to chunks can significantly improve the retrieval quality of RAG systems. This metadata helps the retriever filter chunks and provide richer context to the LLM. For instance, when a user asks, "On which page are the warranty conditions for product X written?", the chunk's metadata including the page number can help the LLM provide a more accurate answer.

Finding the Optimal Chunk Size for Turkish Documents: A Testing Approach

Finding the optimal chunk_size and chunk_overlap values for Turkish documents is not possible with a single magic formula. These values can vary depending on the document type (legal text, technical manual, literary work), its information density, and the embedding model used. Therefore, adopting a systematic testing approach is the most appropriate.

Test Environment and Dataset

To conduct our tests, we will need the following components:

Test Document: A long (e.g., 5-10 pages) Turkish document that reflects a real-world scenario and from which question-answer pairs can be extracted should be selected. User manuals for a production ERP system or a company policy document can be a good starting point.
Embedding Model: An embedding model that understands Turkish well, either multilingual or specifically trained for Turkish (e.g., bert-base-multilingual-cased, T-BERT, mGPT, or models like OpenAI's text-embedding-ada-002).
Vector Database: A vector database (e.g., FAISS, ChromaDB, Pinecone) to store chunks and their embeddings.
LLM: The LLM you plan to use in your RAG system (e.g., OpenAI GPT-4, Gemini Pro, Llama-2).
Question-Answer Pairs: At least 10-20 question-answer pairs manually extracted from the selected test document, covering different difficulty levels (simple information retrieval, inference). This will serve as the gold standard for evaluating retrieval and generation quality.

Example Scenario: Let's consider the Turkish user manual for an ERP system of a manufacturing company. From this manual, a test set is created containing questions like "How is a production order opened?" and "Which menu is used for stock entry?", along with their correct answers from the manual.

Evaluation Metrics

The following metrics can be used to evaluate chunking strategies:

Retrieval Accuracy: Whether the top N chunks retrieved by the retriever for a user query contain the expected correct answer. This indicates how well the retriever is performing.
Answer Relevance: How relevant the answer generated by the LLM is to the user's question and the chunks received from the retriever. This combines both retrieval and generation quality.
Conciseness: Whether the LLM's answer contains unnecessary information or repetitions. Very large chunks can lead to longer answers by carrying extraneous information to the LLM.
Context Utilization: How effectively the LLM uses the information from the chunks retrieved by the retriever.

Example Scenario and Code Snippets

Below is a simple Python-based test flow to try different chunk sizes. This flow will split our defined test document into chunks using different chunk_size and chunk_overlap values, generate their embeddings, and save them to a vector database. We will then manually evaluate the retrieval quality by querying with pre-prepared questions.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OpenAIEmbeddings # Or another multi-lingual model
from langchain_community.vectorstores import Chroma

# 1. Load the Test Document
# Let's use sample text instead of a real file
with open("ornek_turkce_belge.txt", "r", encoding="utf-8") as f:
    turkce_belge_metni = f.read()

# 2. Initialize the Embedding Model
# Add your actual API key or local model here
# Note: The text-embedding-ada-002 model has been updated by OpenAI with text-embedding-3-small and text-embedding-3-large models.
# For new applications, using text-embedding-3-small or text-embedding-3-large is recommended.
# In this example, text-embedding-ada-002 is used to preserve the context of the original article.
embeddings_model = OpenAIEmbeddings(model="text-embedding-ada-002")

# 3. Chunk size and chunk_overlap combinations to test
# We are considering character-based for Turkish.
test_params = [
    {"chunk_size": 250, "chunk_overlap": 50},
    {"chunk_size": 500, "chunk_overlap": 100},
    {"chunk_size": 750, "chunk_overlap": 150},
    {"chunk_size": 1000, "chunk_overlap": 200},
]

results = {}

for params in test_params:
    chunk_size = params["chunk_size"]
    chunk_overlap = params["chunk_overlap"]
    print(f"\n--- Testing with chunk_size={chunk_size}, chunk_overlap={chunk_overlap} ---")

    # Splitting into chunks with RecursiveCharacterTextSplitter
    text_splitter = RecursiveCharacterTextSplitter(
        separators=["\n\n", "\n", " ", ""],
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        length_function=len,
        is_separator_regex=False,
    )
    chunks = text_splitter.split_text(turkce_belge_metni)
    print(f"Total number of chunks: {len(chunks)}")

    # Saving to vector database (new DB for each test)
    vectorstore = Chroma.from_documents(chunks, embeddings_model, collection_name=f"turkce_rag_test_{chunk_size}_{chunk_overlap}")

    # Test questions
    test_questions = [
        "What are the steps to create a production order?",
        "How is stock entry performed?",
        "How is a supplier invoice approved?",
        "Where can I access the reporting screens?",
    ]

    total_relevant_chunks = 0
    correct_chunk_count = 0

    for question in test_questions:
        # Retrieve relevant chunks with the retriever
        retrieved_chunks = vectorstore.similarity_search(question, k=3) # Get the top 3 most relevant chunks

        print(f"\nQuestion: {question}")
        print("Retrieved Chunks:")
        for i, chunk in enumerate(retrieved_chunks):
            print(f"  {i+1}. Chunk ({len(chunk.page_content)} characters):\n  {chunk.page_content[:200]}...\n")
            # Here we need to manually evaluate: Does this chunk contain the answer to the question?
            # In a real system, evaluation would be done with an LLM.
            # For now, a simple check: Is there a relevant keyword for the "production order" question?
            if "üretim emri" in question.lower() and "üretim emri" in chunk.page_content.lower():
                correct_chunk_count += 1
            elif "stok girişi" in question.lower() and "stok girişi" in chunk.page_content.lower():
                correct_chunk_count += 1
            # Similar checks can be added for other questions.
            total_relevant_chunks += 1

    # Simple retrieval accuracy estimation
    if total_relevant_chunks > 0:
        results[f"{chunk_size}-{chunk_overlap}"] = correct_chunk_count / total_relevant_chunks
    else:
        results[f"{chunk_size}-{chunk_overlap}"] = 0

print("\n--- Test Results (Simple Accuracy Estimation) ---")
for config, accuracy in results.items():
    print(f"Configuration {config}: Accuracy = {accuracy:.2f}")

This code snippet demonstrates the steps of chunking with different chunk_size and chunk_overlap values, generating embeddings, and saving them to a vector database. For each configuration, it retrieves the k most relevant chunks using pre-defined test questions. The remaining part is the manual inspection of the retrieved chunks and assigning a score.

⚠️ Manual Evaluation is Critical

The correct_chunk_count calculation in the code above is very basic and relies on keyword matching. In a real scenario, whether the retrieved chunks actually contain the answer to the question, or how accurately the LLM can generate an answer with these chunks, should be evaluated manually or with more sophisticated metrics (e.g., RAGas).

Findings and Solution Recommendations

Based on the tests conducted and field experience, the optimal chunking sizes for Turkish documents are generally more flexible and dependent on the text type compared to English.

Optimal chunk_size: Character-based chunk sizes between 500 and 1000 characters offer a good starting point for most Turkish documents. This range strikes a balance between preserving sufficient context and minimizing extraneous information. When considered in terms of tokens, this might correspond to a range of 128 to 256 tokens, but one must pay attention to how tokenization is performed due to Turkish's agglutinative nature.
Critical chunk_overlap: The chunk_overlap value is particularly important in Turkish texts. An overlap of 100 to 200 characters generally yields good results for preserving semantic transitions between sentences. This ensures that information at the end of one chunk is carried over to the beginning of the next, preventing loss.
Special Cases: Especially in structured documents containing step-by-step instructions or bulleted lists, standard chunkers might cut lists in half, disrupting semantic integrity. In such cases, it might be necessary to add custom separators (e.g., list item markers like "1.", "2.") to the separators list of RecursiveCharacterTextSplitter or try smaller chunk_size and higher chunk_overlap.

Generally, to preserve semantic integrity in Turkish texts, an approach using hierarchical separators like RecursiveCharacterTextSplitter should be preferred. Starting with a chunk_size of 500 characters and a chunk_overlap of 100 characters and then increasing or decreasing these values based on retrieval results is a sensible approach.

Advanced Optimizations and Considerations

Optimal chunking is a continuous optimization process, and more advanced techniques can further enhance the performance of Turkish RAG systems.

Customized Text Splitters

Using text splitters specifically designed for the morphological structure of Turkish can lead to more accurate semantic boundary detection. For example, by leveraging the sentence or paragraph splitting capabilities of natural language processing (NLP) libraries (e.g., Zemberek, Spacy Turkish models), it is possible to divide text into more grammatically consistent segments. This can be particularly useful for legal or academic texts with complex sentence structures.

Combination of Multiple Chunking Strategies

A single chunking strategy may not always yield the best results. Combining multiple strategies that create chunks of different sizes (e.g., both small, detail-oriented chunks and larger, context-oriented chunks) can enable the retriever to better respond to different types of queries. This approach can be implemented with architectures like "small-to-large chunking" or "parent document retriever."

Cross-Encoder Re-ranking

Re-ranking the top k chunks retrieved by the retriever using a cross-encoder model before sending them to the LLM can significantly improve retrieval quality. Cross-encoders evaluate the query and each chunk together to more accurately estimate their semantic relevance. Finding and integrating suitable cross-encoder models for Turkish will directly impact RAG performance.

Security and Performance Trade-offs

More complex chunking strategies and re-ranking steps can increase the system's overall latency and computational costs. Especially in real-time RAG applications, these trade-offs must be carefully evaluated. For instance, a more detailed chunking strategy might offer better accuracy but push the response time to unacceptable levels. In such cases, optimizations (e.g., caching, asynchronous operations) must be made to balance performance and accuracy.

🔥 Cost and Complexity

Every additional optimization step increases the system's cost and maintenance complexity. Therefore, the benefits of each implemented technique must be carefully weighed against its costs. Sometimes, a simpler RecursiveCharacterTextSplitter with well-tuned parameters can yield sufficiently good results compared to an expensive and complex solution.

Conclusion

When working with Turkish documents in RAG systems, the importance of the chunking strategy cannot be overstated. The agglutinative nature of Turkish can cause standard English-centric approaches to fall short. Therefore, using flexible tools like RecursiveCharacterTextSplitter and finding optimal values by testing chunk_size and chunk_overlap parameters is a critical step.

chunk_size values between 500-1000 characters and chunk_overlap values of 100-200 characters generally offer a good starting point. However, these values can differ for each document type and use case. Systematic testing, manual evaluation, and perhaps advanced optimizations (custom splitters, re-ranking) are key to improving the accuracy and efficiency of your Turkish RAG applications. Remember, the best chunking strategy is the one that best suits your application's specific needs and data structure.

DEV Community