RAG Architecture Beyond the Demo: Retrieval, Thai Chunking, and Production Boundaries

Connecting an LLM to an application is the easy part of a RAG demo.

The difficult part starts when the source is a real document instead of a clean string in an array.

I learned this while feeding a Thai tourism PDF into a document pipeline. (You can read more about that story here: "Why Markdown Is the Ultimate AI-Native File Format A War Story from Building NEXT4I" https://www.next4i.com/dev-notes/en/markdown-the-ultimate-ai-native-dev-en).

Direct text extraction produced fragmented characters and misplaced vowels or tone marks. Rendering pages as images preserved the layout but introduced colorful backgrounds and watermarks. Converting them to black and white made some text clearer while removing useful context from charts and photographs. Different models produced different interpretations, so the outputs still required synthesis and human review.

Think of it like OCR processing for an ID card selfie: even with modern vision models, apps still often need human input fields to verify or correct the data. Document ingestion is just step one of RAG, and this is where the real world begins.

That experience changed how I look at Retrieval-Augmented Generation. RAG is not only a prompt pattern. It is a data and retrieval system with an LLM at the end.

This article unpacks that system through:

• separate ingestion and query pipelines
• document extraction and chunking
• embeddings and retrieval
• dense, sparse, and hybrid search
• access control in the retrieval path
• evaluation boundaries
• a conceptual implementation in Python

The code below is generic and sanitized. It demonstrates mechanics, not a production-ready application. Provider names, model IDs, SDK calls, and executable integration still need verification against the versions you use.

The three limits RAG is trying to address

When an application uses an LLM out of the box, three limits appear quickly.

Knowledge boundaries

The model does not automatically know a new event, a private company policy, or a document created yesterday.

Context and cost boundaries

Passing an entire document collection in every request is usually impractical. A large context also does not guarantee that the model will use every passage equally well.

Verifiability

A generated answer does not prove itself. If a user cannot trace a statement back to a permitted source, confidence is not evidence.

RAG addresses these limits by finding a small set of relevant passages and placing them in the context before generation. It can improve grounding and traceability, but it does not guarantee either accuracy or security.

RAG is two pipelines, not one request

A useful mental model separates offline ingestion from online querying.

[Ingestion pipeline]

Raw documents
    |
    v
Extract and clean
    |
    v
Split into chunks
    |
    v
Create embeddings and metadata
    |
    v
Store in a searchable index


[Query pipeline]

User question
    |
    v
Apply identity and permission scope
    |
    v
Retrieve candidate chunks
    |
    v
Filter and optionally rerank
    |
    v
Build prompt with citations
    |
    v
Generate answer
    |
    v
Return answer and inspectable sources

The pipelines are decoupled for a reason. Documents change on their own schedule. User queries arrive on another. Extraction failures should not be rediscovered during every chat request, and permission changes should not require retraining a model.

Ingestion quality becomes retrieval quality

The phrase “garbage in, garbage out” is almost too familiar, but it describes RAG accurately.

A document can look correct to a human and still become poor retrieval material after extraction. Typical problems include:

• repeated headers and footers
• tables flattened into an unreadable order
• text stored as positioned glyphs rather than paragraphs
• watermarks mixed with body text
• diagrams whose meaning exists only in the visual layout
• scanned pages without a reliable text layer
• old and current versions indexed together

The ingestion pipeline needs to preserve more than plain text. Useful metadata can include a stable document ID, version, page, section heading, timestamps, source location, tenant, and permission attributes.

That metadata supports citations, updates, deletion, filtering, and review. Without it, a retrieved paragraph becomes an orphan.

Thai documents make naive chunking easier to break

Many chunking examples assume that spaces and punctuation provide reliable boundaries. Thai text does not always behave that way.

Ambiguous boundaries

Consider the Thai text:

ตากลมนั่งมองตากลม

Depending on segmentation and context, ตากลม can relate to round eyes, while ตากลม can also be read through a boundary involving exposure to wind. A bad split changes the meaning represented by the chunk and therefore affects retrieval.

Long dependencies

A policy sentence may introduce the subject and action early, then place the condition or consequence much later. A fixed-size cut can leave one chunk with the violation and another with the disciplinary action. Neither passage is complete enough to answer the question reliably.

Periods that are not sentence endings

Thai abbreviations, titles, legal terms, locations, and time expressions can contain periods:

ศ.ดร.สมชาย ... พ.ร.บ. ... อ.เมือง จ.เชียงใหม่ ... 09.00 น.

A generic splitter that treats every period as an end of sentence can create tiny, meaningless chunks.

Possible approaches from the original engineering notes include:

• semantic chunking around topic changes
• Thai-aware tokenization, for example with PyThaiNLP, plus a domain dictionary
• overlap around chunk boundaries
• hierarchy-aware parent and child chunks

The correct strategy and parameters are corpus-specific. Any fixed chunk size or overlap percentage should be evaluated against real questions rather than copied as a universal setting.

Chunking strategies and their trade-offs

Fixed size with overlap

This is simple and predictable. It also cuts across structure when the chosen length does not match the document.

Recursive splitting

This method tries larger structural separators first, such as sections and paragraphs, then falls back to smaller boundaries. It works better when the extracted structure is trustworthy.

Semantic chunking

This approach compares nearby passages and creates a new chunk when the topic shifts. It can preserve meaning better, but it adds model dependency, threshold tuning, and processing cost.

Parent-child chunking

Small child chunks can support precise retrieval while a larger parent section supplies enough context to answer. The trade-off is more complex indexing, deduplication, and prompt assembly.

Chunking should be treated as a retrieval decision, not a formatting task. The unit you retrieve determines the evidence the model can see.

Embeddings are coordinates, not facts

An embedding converts content into a numeric vector. Content with related meaning may appear near each other in that vector space, depending on the model and data.

That makes semantic retrieval possible, but an embedding does not verify truth. It also does not know that an older policy is invalid unless version and filtering logic provide that boundary.

Conceptual implementation in Python (In-Memory Retrieval & Query Pipeline)

To illustrate how a query pipeline works in one place, the following Python example simulates cosine similarity calculation, user permission and tenant filtering prior to retrieval, top-K ranking, and prompt augmentation for the LLM:

import math

def cosine_similarity(vec_a, vec_b):
    """Compute cosine similarity between two numeric vectors."""
    if len(vec_a) != len(vec_b) or len(vec_a) == 0:
        raise ValueError("Vectors must have the same non-zero length")

    dot_product = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))

    if norm_a == 0 or norm_b == 0:
        raise ValueError("Cosine similarity is undefined for a zero vector")

    return dot_product / (norm_a * norm_b)

def answer_with_rag(question, user, knowledge_base, top_k=2, create_embedding=None, generate_answer=None):
    """Demonstrate an end-to-end RAG query flow with permission scoping."""
    # 1. Convert user question into an embedding vector
    query_vector = create_embedding(question)

    # 2. Apply identity and permission scope before retrieval
    # Critical: The model must never receive context the user is not allowed to see
    authorized_chunks = [
        chunk for chunk in knowledge_base
        if chunk["tenant_id"] == user["tenant_id"] and user["role"] in chunk["allowed_roles"]
    ]

    # 3. Retrieval: calculate similarity and rank Top-K
    ranked_chunks = [
        {**chunk, "score": cosine_similarity(query_vector, chunk["embedding"])}
        for chunk in authorized_chunks
    ]
    ranked_chunks.sort(key=lambda item: item["score"], reverse=True)
    selected_chunks = ranked_chunks[:top_k]

    # 4. Augmentation: build context with verifiable source citations
    context_text = "\n\n".join([f"[ID: {c['id']}] {c['content']}" for c in selected_chunks])

    # 5. Generation: ask the model to ground its response in the provided context
    instruction = "Answer only from the supplied context. If the context is insufficient, state that clearly."
    answer = generate_answer(
        instruction=instruction,
        question=question,
        context=context_text
    )

    return {
        "answer": answer,
        "sources": [{"id": c["id"], "score": round(c["score"], 4)} for c in selected_chunks]
    }

The core takeaway is that permission filtering must happen before passages become model context. In production enterprise systems, this is handled through row-level security (RLS), policy engines, document ACLs, or metadata filtering on a vector store rather than an in-memory loop.

Dense search is not enough for every query

Dense vector search is useful for semantic similarity. It can connect related wording such as synonyms and paraphrases.

Exact identifiers are a different problem. Product SKUs, serial numbers, legal section numbers, error codes, and names can be better served by sparse keyword retrieval.

That creates three common choices:

• Dense retrieval: strong for semantic similarity
• Sparse retrieval such as BM25: strong for exact terms
• Hybrid retrieval: combines candidates from both systems, then fuses or reranks them

Reciprocal Rank Fusion and cross-encoder reranking are possible tools in that pipeline. They are not automatic improvements for every corpus. Evaluate them using your own question set and latency boundary.

Five production boundaries worth testing

1. Language and embedding fit

Do not assume an embedding model that looks good on English examples will behave the same way on Thai policies, abbreviations, and domain terms. Compare models against labeled queries from the actual corpus.

2. Extraction and version quality

Track failed pages, missing sections, table quality, repeated boilerplate, and document versions. Retrieval cannot recover content that ingestion lost.

3. Context pollution

More chunks are not always better. Irrelevant context can distract the model and increase cost. Evaluate the complete answer path rather than maximizing topK by intuition.

4. Security and data isolation

Permission-aware retrieval needs to be enforced outside the model. Test cross-tenant access, role changes, deleted permissions, cached results, and citation links.

5. Evaluation

The original notes use three useful questions:

• Context relevance: Did retrieval find evidence related to the question?
• Groundedness or faithfulness: Does the answer stay within the supplied evidence?
• Answer relevance: Does it answer what the user asked?

Add operational checks that match the system, such as extraction failures, stale indexes, permission denials, and citation integrity. Do not invent target scores. Establish them from your risk and user requirements.

The main engineering lesson

RAG connects language-model capability with external knowledge, but the connection is only as useful as the pipeline around it.

A local demo can hide document quality, authorization, stale versions, and evaluation because the sample chunks are already clean. Production exposes all of them.

For me, the work starts before prompt engineering:

1. prepare and version the knowledge
2. choose chunk boundaries that preserve meaning
3. retrieve with semantic and exact-match needs in mind
4. apply permissions before context construction
5. preserve citations
6. evaluate retrieval and generation separately

The LLM writes the final response. Most of the evidence it can use has already been decided by then.

---

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en