DEV Community: NEXT4I DEV

Chunking in RAG: How to Split Documents Without Losing the Conditions

NEXT4I DEV — Wed, 07 Oct 2026 18:27:50 +0000

Your AI assistant has the employee benefits handbook. You ask: “Can someone who is still on probation claim the eyewear benefit?”

It confidently answers: “Yes, up to 3,000 baht per year.” But the handbook says the benefit is only for permanent employees who have completed probation. The limit reached the model. The eligibility condition did not.

That failure can start before generation, at the document boundary. Chunking in RAG is not about making text as small as possible. It is about creating searchable pieces without separating facts from the conditions that make them true.

The policy and amounts here are fictional examples, not NEXT4I or customer policies.

TL;DR

A chunk is a piece of source content used for retrieval. It is not inherently a vector.
Large chunks can mix unrelated topics. Small chunks can lose subjects, conditions, and exceptions.
Start with document structure, cap oversized sections, and test with real questions.
Overlap and parent-child retrieval can help with missing context. They cannot repair incorrectly extracted text.
Check retrieved evidence, context completeness, and answer correctness separately.

What is chunking, and why not use the entire file?

Chunking divides a document into pieces that can be indexed and retrieved. A simplified pipeline looks like this:

Document -> Extract and organize text -> Create chunks
         -> Store text and source references -> Build an index

Question -> Retrieve relevant chunks -> Assemble context -> LLM answer with sources

Retrieval may use keyword search, vector search, or both. Not every chunk needs an embedding. When vector search is used, an embedding model creates a numerical representation of the content.

The searchable unit also does not have to be the final reading unit. A system can retrieve a short passage, then supply its surrounding section to the model.

A single chunk containing a 100-page handbook mixes many topics. That can make specific passages harder to retrieve and exceed model input limits. Sending unrelated material also costs tokens and can distract from the evidence that matters.

Whole-document context can work for short documents. Tiny chunks are not automatically better: an amount without its eligibility rule is incomplete evidence.

One boundary can change the answer

Consider this fictional passage:

Section 8.1: Prescription eyewear benefit
The company reimburses up to 3,000 baht per year.
Only permanent employees who have completed probation are eligible.

A length-based split might produce:

Chunk A: The company reimburses up to 3,000 baht per year.
Chunk B: Only permanent employees who have completed probation are eligible.

Without the heading, both pieces lose their subject. B never names the benefit, and vector search does not guarantee recovery of that missing relationship.

If only A reaches the LLM, it has evidence about the amount, not enough evidence about eligibility. A prompt telling it not to invent information may help it acknowledge uncertainty. It cannot restore a sentence that was never supplied.

Keeping the heading, amount, and condition together helps make evidence usable. It does not guarantee a correct answer.

The main approaches and their trade-offs

These approaches can be combined. Splitting by heading and then dividing oversized sections is often more useful than committing to one method everywhere.

1. Fixed-size splitting

Choose a length, such as 500 tokens, and divide the text into consecutive pieces. It is straightforward, but the boundary may cut through a sentence, table, or exception.

Use the actual model tokenizer when token limits matter. Character counts are not interchangeable with token counts, and a Thai word is not necessarily one token.

2. Fixed-size splitting with overlap

Overlap repeats some text from one chunk in the next, giving material near the boundary a chance to stay together.

It cannot fix distant conditions or scrambled reading order. Thai OCR with unreliable spacing still needs boundary checks.

More overlap means more indexed content and duplication. Near-identical results can occupy retrieval slots that should contain other evidence. Deduplicate or merge overlapping context before sending it to the LLM where appropriate.

3. Recursive splitting

A recursive splitter tries separators in a chosen order, such as paragraphs, sentences, then finer boundaries, until pieces fit the size limit.

It avoids unnecessary cuts through sentences, but does not understand meaning. A paragraph can cover several topics; damaged OCR can remove boundaries.

4. Structure-aware splitting

Use headings, lists, tables, and code blocks to guide boundaries. A subsection can carry its parent heading so a short passage still has a clear subject.

Tables need special care. An isolated “3,000” is not enough. Preserve the row name, column name, and unit. In our fictional example, “Prescription eyewear: annual reimbursement limit of 3,000 baht” is clearer. Any eligibility note must accompany it too.

Code and ordered requirements have similar dependencies. If an oversized block must be split, preserve its relationships and necessary conditions.

5. Semantic chunking

Semantic methods estimate where nearby sentences or paragraphs change topic, often using embeddings or a model.

They can help when structure is unclear, but add processing cost and tuning. Results depend on the model and source quality. If headings already separate topics well, I would test that simpler structure first.

6. Parent-child retrieval and contextual chunking

These solve different problems alongside the splitting methods above.

Parent-child retrieval indexes small child passages for search, then expands a match to a larger parent section for reading. Maintain the links, control context size, and check access permissions for the expanded parent too.

Contextual chunking adds useful context before indexing, such as a document title, section name, or short source-grounded explanation. If an LLM generates that explanation, check that it has not invented conditions. Keep original text available for citations instead of treating generated context as source evidence.

Propositional or agentic approaches may use an LLM to extract standalone statements or choose content groupings. Implementations vary. The shared trade-offs are extra cost and the risk that rewriting drops a qualification. Compare them with a non-LLM baseline and retain the original source.

Match the approach to the document

Document	A reasonable first approach	Check carefully
Markdown or structured manuals	Split by heading, subdivide oversized sections	Parent headings, tables, footnotes
Plain text with inconsistent structure	Recursive splitting, limited overlap	Conditions at boundaries
Contracts or rules requiring several paragraphs	Retrieve small passages, expand or combine evidence	Clause numbers, versions, permissions
Scanned or multi-column PDFs	Repair extraction and reading order first	OCR errors, scrambled columns, broken tables

These are starting points, not substitutes for inspecting actual documents.

The Thai tourism PDF that changed my priorities

While experimenting with a RAG pipeline, I used a beautifully designed tourism PDF about Thailand. The extracted text was much less attractive: Thai vowels and tone marks were misplaced, while colored backgrounds, watermarks, and illustrations interfered with reading.

For some pages, I tried black-and-white conversion to make the letters clearer. That could also lose visual context. I compared outputs with help from multiple models and had a person review spelling again.

I described this in my earlier PDF and Markdown write-up, in Thai. It is one experience, not evidence that every PDF needs the same workflow.

The lesson was simple: changing chunk size cannot bring back words that extraction has already lost.

Markdown makes structure easier to identify, not evaluation unnecessary. Sections can still be oversized, and table rows need headers.

How large should chunks and overlap be?

There is no universal setting. 300-500 tokens with 10-20% overlap can be an initial experiment for paragraph-based text. Those numbers are not a benchmark result from my system or a standard to apply everywhere.

Before tuning, check four things:

Meaning: Does each piece identify the subject and preserve necessary conditions?
Limits: Use the actual tokenizer and budget for instructions, the question, all retrieved context, and the answer.
Duplication: Does overlap add useful evidence or just repeat it?
Question type: A lookup for an amount and a question about eligibility may require different evidence.

Start with structure and a size cap. Add overlap or parent expansion when tests reveal missing context, not simply because more context feels safer.

How to tell whether chunking works

Build questions with known supporting documents, sections, or pages. Include boundary-sensitive questions, amount lookups, and questions the documents cannot answer.

Compare two or three approaches on the same sources and questions with comparable context budgets. Then inspect three layers:

Retrieval: Are the labeled supporting passages found? Recall@k measures retrieval of predefined relevant evidence in the top k results. Separately check whether the evidence available is sufficient for the question.
Context: Are conditions and exceptions present? Do duplicates crowd out other evidence? Are table headers and units intact?
Answer: Does it follow the evidence? Does each citation support the associated claim? Can the system acknowledge insufficient information?

Store metadata such as document_id, section, page/offset, and version so passages can be traced and stale index entries managed.

For organizational data, enforce permissions before content reaches the model, including any expanded parent context. Hiding the final answer is not a substitute for controlling what the model receives.

If source text is wrong, fix ingestion. If evidence exists but is not found, inspect chunking and retrieval. If complete evidence reaches the model but the answer is wrong, inspect context assembly and generation.

The goal is usable evidence

A RAG system generally supplies selected evidence, not the entire library, for each question. Chunk boundaries help determine what the model gets to see, though they are not the only factor in answer quality.

My starting point is accurate source text, structure-aware boundaries, sensible size limits, source references, and tests with real questions. Add complexity where those tests show a need.

The best chunk is not necessarily the smallest one. It is a piece the system can find and use without losing what makes the answer valid.

If you are building RAG, share which document types cause missing conditions and where in the pipeline you catch them.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

From Ctrl + F to AI-Powered Search

NEXT4I DEV — Wed, 30 Sep 2026 18:11:58 +0000

How Do Computers Find Content and Answer Questions from Files?

(A Deep Dive into Search & Retrieval Fundamentals: From Ctrl+F to AI Semantic Search)

In the daily work of modern tech teams and knowledge workers, there is a classic problem that never really goes away:

"We know this information exists somewhere, but we have no idea which file, which page, or which line it is on."

In the past, we relied on Ctrl + F to search through documents one file at a time.
Later, search servers and file search tools introduced indexing, allowing us to find file names and keywords much faster across storage systems.
Today, in the era of Generative AI, we can simply ask a question in a chat interface: "Summarize the vendor contract for me: what is the late delivery penalty?" And within seconds, the system digs through a 200-page document and extracts the exact numbers and contractual conditions.

Behind the scenes, though, how does a computer actually "search" and "understand" the content inside our files?

Why does traditional keyword search frequently miss critical answers? And why do modern search engines seem to understand what we mean, even when our queries share zero words with the source document?

This article walks through the engineering foundations of Search & Retrieval, from brute-force byte scans to vector embeddings and AI-powered synthesis. It provides the essential mental model you need before diving deeper into Retrieval-Augmented Generation (RAG).

1. The Fundamental Gap: Humans Read Meaning, Computers Read Bytes

Before discussing search algorithms, we need to understand the fundamental limitation of computers:

Humans: When we read the phrase "an employee is unwell", we instantly connect it with "sick leave", "medical certificate", "hospital", or "health insurance benefits".
Computers: At a baseline level, the phrase "sick leave" is nothing more than an array of UTF-8 bytes. The computer does not inherently know that this sequence of bytes relates to physical health, time off work, or payroll deductions.

This gap created two distinct eras of information retrieval:

Lexical Search: Cares strictly about character matching: "Are the words spelled the same way?"
Semantic Search: Cares about conceptual intent: "Are the meaning and context aligned?"

2. Era 1: Exact Character Matching (Lexical / Exact Keyword Search)

This is where search technology started. In production environments, it primarily takes two forms:

2.1 Linear Scan (`Ctrl + F` or `grep`)

How it works: When a user searches for loan interest, the system opens the file from the very first byte and iterates through every line sequentially until it reaches the end.
Complexity: $O(N)$, where $N$ is the total number of characters or bytes in the dataset.
Advantages:
- Real-time accuracy: if you edit a file, the next search reflects the update immediately without an indexing delay.
- Zero storage overhead: no dedicated search database or pre-computed structures are needed.
Disadvantages:
- Does not scale: If you have 10,000 files with hundreds of millions of words, opening and scanning every file sequentially will quickly exhaust CPU cycles and disk I/O.

2.2 Inverted Index (The Backbone of Search Engines)

To eliminate the bottleneck of linear scans, computer scientists developed the Inverted Index, which powers engines like Elasticsearch, Apache Lucene, and OpenSearch:

How it works: Instead of waiting for a query before opening files, the system analyzes and indexes documents ahead of time (Index Time). The indexing pipeline reads the documents, splits text into tokens (Tokenization), removes irrelevant filler words (Stopwords), and records a lookup table: "Which files and positions contain this specific word?", much like the index at the back of a textbook.

[ Source Documents ]
File_A: "Employee is entitled to sick leave for 30 days"
File_B: "Submitting sick leave requires a medical certificate"

[ Inverted Index Table ]
Term                 │ Posting List (Occurrences)
─────────────────────┼─────────────────────────────────────────────
certificate          │ File_B (pos 7)
days                 │ File_A (pos 8)
employee             │ File_A (pos 1)
leave                │ File_A (pos 5), File_B (pos 3)
medical              │ File_B (pos 6)
sick                 │ File_A (pos 4), File_B (pos 2)

At query time (Query Time): When a user searches for "sick leave", the system does not rescan File A or File B from scratch. It directly looks up the table and returns matching documents in sub-millisecond time ($O(1)$ or $O(\log M)$, where $M$ is the vocabulary size of the index).

Limitations of Lexical Search: Why Exact Matching Falls Short

While inverted indexes are fast and power production systems through algorithms like BM25, exact character matching suffers from three inherent language barriers:

The Synonym Problem: If a policy document says "Rules for absence while undergoing medical treatment" but the user searches for "how to take sick leave", the system returns zero results, even though the content is a 100% match in meaning.
The Polysemy Problem (Word Ambiguity): The same word can carry completely different meanings depending on context. For example, searching for "crane" might return documents about construction machinery, a bird species, or an industrial pulley system without distinction.
Typos & Natural Questions: A single typo can break a match. Furthermore, when users ask conversational questions like "If someone gets burned in the cafeteria kitchen, who handles the medical claim?", tokenizers break the query into fragmented words, which distorts keyword relevance scores.

3. Era 2: Searching by Meaning (Semantic / Vector Search)

To overcome the limitations of spelling, modern AI leverages deep learning to create Text Embeddings and Vector Search.

In early hands-on experiments, when seeding test data with different wording, paraphrased sentences, or even cross-language queries, running a top-k similarity search using embedding models like BAAI/bge-m3 consistently retrieved documents with matching intent, despite having zero exact word overlap.

3.1 What is Text Embedding? (Turning Language into Coordinates)

For those interested in exploring how text embeddings power Retrieval-Augmented Generation, check out NEXT4I's Guide on RAG.

An embedding model (such as OpenAI's text-embedding-3, Cohere Embed, or open-source models like BAAI/bge) takes text passages as input and maps them into a high-dimensional dense vector, such as 768 or 1,536 dimensions.

The core principle:

"Passages with similar meaning or shared context end up close together as clusters in this multi-dimensional vector space."

                     [ Semantic Cluster: Illness & Healthcare ]
                     • "Employee is feeling sick"
                     • "Medical leave guidelines"
                     • "Healthcare expense claims"

  [ Semantic Cluster: Finance ]              [ Semantic Cluster: Time Off ]
  • "Q3 balance sheet review"                • "Submitting annual leave"
  • "Withholding tax deduction"              • "Official public holidays"

3.2 Measuring Proximity Mathematically (Similarity Metrics)

Once texts are converted into vectors, a computer determines conceptual similarity through geometry, most commonly via Cosine Similarity:

$$\text{Cosine Similarity}(\vec{A}, \vec{B}) = \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|}$$

Value close to 1.0: The vectors point in nearly identical directions, indicating closely aligned conceptual meaning.
Value close to 0: The vectors are orthogonal, indicating little to no contextual correlation.

Example:

Query: "Caught the flu, who should I notify?" $\rightarrow$ Vector $\vec{Q}$
Document A: "Procedure for submitting doctor's note to team lead" $\rightarrow$ Vector $\vec{D_1}$ (Cosine Similarity = 0.88)
Document B: "Setting up email passwords for sales department" $\rightarrow$ Vector $\vec{D_2}$ (Cosine Similarity = 0.12)

The search system immediately retrieves Document A because its cosine similarity is exceptionally high, even though Document A contains neither the word "flu" nor "notify".

4. Why Not Convert an Entire 100-Page File into a Single Vector?

A frequent question is: "If vector search is so powerful, why not embed an entire 100-page PDF into one vector and be done with it?"

Doing so fails in production for three key engineering reasons:

Information Dilution: If you take an employee handbook covering procurement, vacation policies, legal compliance, IT security, and annual bonuses, and compress all 100 pages into a single vector, that coordinate becomes a vague mathematical average of all topics. When a user asks a specific question, the combined vector will sit far away from the query. It is like blending dozens of different fruits into one container: you end up with a diluted mixture where individual flavors are unrecognizable.
Context Window Limits of Embedding Models: Embedding models have token input limits, such as 512, 2,048, or 8,192 tokens. They cannot ingest megabytes of raw text in a single pass.
Precision for Answer Generation: When retrieving evidence, we need to pass only the specific, relevant passages into the language model. Feeding 100 pages of irrelevant noise increases latency, cost, and hallucination risks.

The Solution: The Art of Chunking (Chunking Strategy)

Before indexing documents, we split files into smaller passages (chunks):

Typically 300 to 500 words per chunk.
A slight overlap (e.g., 50 words) to prevent context from breaking across chunk boundaries.
Each chunk maintains a focused conceptual topic and receives its own unique vector representation.

5. From Finding Content to Answering Questions: The 3-Step Pipeline

Search and retrieval systems only identify where the evidence lives. In real-world workflows, users rarely want ten raw links or disconnected paragraphs: they want a concise, reliable, and actionable answer.

This is where Retrieval connects with Generative AI (LLMs) through a three-step pipeline:

[ 1. User Asks a Question ]
  "Can an employee who is still on probation claim eyeglasses reimbursement?"
       │
       ▼
[ 2. Step 1: RETRIEVAL (Locate Relevant Chunks) ]
  - The system converts the user's question into a query vector.
  - Performs an ANN search across a Vector Database or Inverted Index.
  - Retrieves the top-k chunks with the highest relevance scores.
       │
       ▼
[ 3. Step 2: CONTEXT AUGMENTATION (Assemble Prompt for AI) ]
  - Combines the user query with the retrieved evidence passages:

    ┌────────────────────────────────────────────────────────┐
    │ SYSTEM: Answer the question using ONLY the provided    │
    │ context. Do not extrapolate. If the context does not   │
    │ contain the answer, reply with "Information not found".│
    │                                                        │
    │ CONTEXT:                                               │
    │ [Chunk #42]: Section 8.1 Eyeglasses Benefit            │
    │ The company provides an annual allowance of $100 for   │
    │ prescription glasses. This benefit is reserved exclusively│
    │ for permanent full-time employees who have successfully│
    │ passed their probationary period...                    │
    │                                                        │
    │ QUESTION: Can an employee on probation claim glasses?  │
    └────────────────────────────────────────────────────────┘
       │
       ▼
[ 4. Step 3: GENERATION & REASONING (Synthesize the Answer) ]
  - The LLM reads only the verified context provided.
  - Formulates a concise answer with source citation:

    👉 "No, employees on probation are not eligible. The optical
        allowance ($100/year) is reserved exclusively for permanent
        employees who have completed probation (Ref: Section 8.1)."

6. Comparison: 4 Paradigms of Document Retrieval

Here is a side-by-side comparison of how search methods have evolved:

Dimension	Linear Scan (`Ctrl+F` / `grep`)	Inverted Index (BM25 / Search Engines)	Semantic Vector Search (Embeddings)	Retrieval + LLM (RAG Paradigm)
Core Mechanism	Scans and compares characters line by line	Looks up pre-compiled word dictionary and postings	Measures geometric distance between dense vectors	Retrieves matching evidence chunks, then asks AI to synthesize
Search Speed	Slows down linearly with data size ($O(N)$)	Sub-millisecond lookup	Fast using Approximate Nearest Neighbor (ANN/HNSW)	Moderate (depends on LLM inference time)
Intent Understanding	None	None (tracks term frequency and statistics)	Understands semantic concepts and context	Deep contextual reasoning with synthesis
Handling Synonyms & Typos	Fails completely	Limited (Fuzzy matching, lemmatization)	Excellent	Best-in-class
Output Format	Line positions or character highlights	Document IDs and line offsets	Relevant text chunks	Direct synthesized answer with references

7. Conclusion: What Should Real-World Systems Use?

When building production document intelligence systems, we do not have to pick one method and discard the rest:

If you rely solely on Keyword Search, you will miss documents whenever user terminology diverges from source wording.
If you rely solely on Vector Search, you may struggle with exact identifiers such as SKU numbers (NK-9920-X), tax IDs, version strings, or domain-specific jargon absent from pre-trained embeddings.

This is why modern enterprise search architectures adopt Hybrid Search, combining lexical keyword indexes with vector embeddings, running the candidates through a Re-ranking stage, and passing the most relevant evidence to an LLM.

At NEXT4I, these search principles guide how we explore and architect document intelligence solutions, ensuring retrieval systems deliver speed, accuracy, and provable evidence. Stay tuned for future deep dives as we continue building and sharing our engineering journey.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

RAG Architecture Beyond the Demo: Retrieval, Thai Chunking, and Production Boundaries

NEXT4I DEV — Wed, 23 Sep 2026 15:37:19 +0000

Connecting an LLM to an application is the easy part of a RAG demo.

The difficult part starts when the source is a real document instead of a clean string in an array.

I learned this while feeding a Thai tourism PDF into a document pipeline. (You can read more about that story here: "Why Markdown Is the Ultimate AI-Native File Format A War Story from Building NEXT4I" https://www.next4i.com/dev-notes/en/markdown-the-ultimate-ai-native-dev-en).

Direct text extraction produced fragmented characters and misplaced vowels or tone marks. Rendering pages as images preserved the layout but introduced colorful backgrounds and watermarks. Converting them to black and white made some text clearer while removing useful context from charts and photographs. Different models produced different interpretations, so the outputs still required synthesis and human review.

Think of it like OCR processing for an ID card selfie: even with modern vision models, apps still often need human input fields to verify or correct the data. Document ingestion is just step one of RAG, and this is where the real world begins.

That experience changed how I look at Retrieval-Augmented Generation. RAG is not only a prompt pattern. It is a data and retrieval system with an LLM at the end.

This article unpacks that system through:

separate ingestion and query pipelines
document extraction and chunking
embeddings and retrieval
dense, sparse, and hybrid search
access control in the retrieval path
evaluation boundaries
a conceptual implementation in Python

The code below is generic and sanitized. It demonstrates mechanics, not a production-ready application. Provider names, model IDs, SDK calls, and executable integration still need verification against the versions you use.

The three limits RAG is trying to address

When an application uses an LLM out of the box, three limits appear quickly.

Knowledge boundaries

The model does not automatically know a new event, a private company policy, or a document created yesterday.

Context and cost boundaries

Passing an entire document collection in every request is usually impractical. A large context also does not guarantee that the model will use every passage equally well.

Verifiability

A generated answer does not prove itself. If a user cannot trace a statement back to a permitted source, confidence is not evidence.

RAG addresses these limits by finding a small set of relevant passages and placing them in the context before generation. It can improve grounding and traceability, but it does not guarantee either accuracy or security.

RAG is two pipelines, not one request

A useful mental model separates offline ingestion from online querying.

[Ingestion pipeline]

Raw documents
    |
    v
Extract and clean
    |
    v
Split into chunks
    |
    v
Create embeddings and metadata
    |
    v
Store in a searchable index


[Query pipeline]

User question
    |
    v
Apply identity and permission scope
    |
    v
Retrieve candidate chunks
    |
    v
Filter and optionally rerank
    |
    v
Build prompt with citations
    |
    v
Generate answer
    |
    v
Return answer and inspectable sources

The pipelines are decoupled for a reason. Documents change on their own schedule. User queries arrive on another. Extraction failures should not be rediscovered during every chat request, and permission changes should not require retraining a model.

Ingestion quality becomes retrieval quality

The phrase “garbage in, garbage out” is almost too familiar, but it describes RAG accurately.

A document can look correct to a human and still become poor retrieval material after extraction. Typical problems include:

repeated headers and footers
tables flattened into an unreadable order
text stored as positioned glyphs rather than paragraphs
watermarks mixed with body text
diagrams whose meaning exists only in the visual layout
scanned pages without a reliable text layer
old and current versions indexed together

The ingestion pipeline needs to preserve more than plain text. Useful metadata can include a stable document ID, version, page, section heading, timestamps, source location, tenant, and permission attributes.

That metadata supports citations, updates, deletion, filtering, and review. Without it, a retrieved paragraph becomes an orphan.

Thai documents make naive chunking easier to break

Many chunking examples assume that spaces and punctuation provide reliable boundaries. Thai text does not always behave that way.

Ambiguous boundaries

Consider the Thai text:

ตากลมนั่งมองตากลม

Depending on segmentation and context, ตากลม can relate to round eyes, while ตากลม can also be read through a boundary involving exposure to wind. A bad split changes the meaning represented by the chunk and therefore affects retrieval.

Long dependencies

A policy sentence may introduce the subject and action early, then place the condition or consequence much later. A fixed-size cut can leave one chunk with the violation and another with the disciplinary action. Neither passage is complete enough to answer the question reliably.

Periods that are not sentence endings

Thai abbreviations, titles, legal terms, locations, and time expressions can contain periods:

ศ.ดร.สมชาย ... พ.ร.บ. ... อ.เมือง จ.เชียงใหม่ ... 09.00 น.

A generic splitter that treats every period as an end of sentence can create tiny, meaningless chunks.

Possible approaches from the original engineering notes include:

semantic chunking around topic changes
Thai-aware tokenization, for example with PyThaiNLP, plus a domain dictionary
overlap around chunk boundaries
hierarchy-aware parent and child chunks

The correct strategy and parameters are corpus-specific. Any fixed chunk size or overlap percentage should be evaluated against real questions rather than copied as a universal setting.

Chunking strategies and their trade-offs

Fixed size with overlap

This is simple and predictable. It also cuts across structure when the chosen length does not match the document.

Recursive splitting

This method tries larger structural separators first, such as sections and paragraphs, then falls back to smaller boundaries. It works better when the extracted structure is trustworthy.

Semantic chunking

This approach compares nearby passages and creates a new chunk when the topic shifts. It can preserve meaning better, but it adds model dependency, threshold tuning, and processing cost.

Parent-child chunking

Small child chunks can support precise retrieval while a larger parent section supplies enough context to answer. The trade-off is more complex indexing, deduplication, and prompt assembly.

Chunking should be treated as a retrieval decision, not a formatting task. The unit you retrieve determines the evidence the model can see.

Embeddings are coordinates, not facts

An embedding converts content into a numeric vector. Content with related meaning may appear near each other in that vector space, depending on the model and data.

That makes semantic retrieval possible, but an embedding does not verify truth. It also does not know that an older policy is invalid unless version and filtering logic provide that boundary.

Conceptual implementation in Python (In-Memory Retrieval & Query Pipeline)

To illustrate how a query pipeline works in one place, the following Python example simulates cosine similarity calculation, user permission and tenant filtering prior to retrieval, top-K ranking, and prompt augmentation for the LLM:

import math

def cosine_similarity(vec_a, vec_b):
    """Compute cosine similarity between two numeric vectors."""
    if len(vec_a) != len(vec_b) or len(vec_a) == 0:
        raise ValueError("Vectors must have the same non-zero length")

    dot_product = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))

    if norm_a == 0 or norm_b == 0:
        raise ValueError("Cosine similarity is undefined for a zero vector")

    return dot_product / (norm_a * norm_b)

def answer_with_rag(question, user, knowledge_base, top_k=2, create_embedding=None, generate_answer=None):
    """Demonstrate an end-to-end RAG query flow with permission scoping."""
    # 1. Convert user question into an embedding vector
    query_vector = create_embedding(question)

    # 2. Apply identity and permission scope before retrieval
    # Critical: The model must never receive context the user is not allowed to see
    authorized_chunks = [
        chunk for chunk in knowledge_base
        if chunk["tenant_id"] == user["tenant_id"] and user["role"] in chunk["allowed_roles"]
    ]

    # 3. Retrieval: calculate similarity and rank Top-K
    ranked_chunks = [
        {**chunk, "score": cosine_similarity(query_vector, chunk["embedding"])}
        for chunk in authorized_chunks
    ]
    ranked_chunks.sort(key=lambda item: item["score"], reverse=True)
    selected_chunks = ranked_chunks[:top_k]

    # 4. Augmentation: build context with verifiable source citations
    context_text = "\n\n".join([f"[ID: {c['id']}] {c['content']}" for c in selected_chunks])

    # 5. Generation: ask the model to ground its response in the provided context
    instruction = "Answer only from the supplied context. If the context is insufficient, state that clearly."
    answer = generate_answer(
        instruction=instruction,
        question=question,
        context=context_text
    )

    return {
        "answer": answer,
        "sources": [{"id": c["id"], "score": round(c["score"], 4)} for c in selected_chunks]
    }

The core takeaway is that permission filtering must happen before passages become model context. In production enterprise systems, this is handled through row-level security (RLS), policy engines, document ACLs, or metadata filtering on a vector store rather than an in-memory loop.

Dense search is not enough for every query

Dense vector search is useful for semantic similarity. It can connect related wording such as synonyms and paraphrases.

Exact identifiers are a different problem. Product SKUs, serial numbers, legal section numbers, error codes, and names can be better served by sparse keyword retrieval.

That creates three common choices:

Dense retrieval: strong for semantic similarity
Sparse retrieval such as BM25: strong for exact terms
Hybrid retrieval: combines candidates from both systems, then fuses or reranks them

Reciprocal Rank Fusion and cross-encoder reranking are possible tools in that pipeline. They are not automatic improvements for every corpus. Evaluate them using your own question set and latency boundary.

Five production boundaries worth testing

1. Language and embedding fit

Do not assume an embedding model that looks good on English examples will behave the same way on Thai policies, abbreviations, and domain terms. Compare models against labeled queries from the actual corpus.

2. Extraction and version quality

Track failed pages, missing sections, table quality, repeated boilerplate, and document versions. Retrieval cannot recover content that ingestion lost.

3. Context pollution

More chunks are not always better. Irrelevant context can distract the model and increase cost. Evaluate the complete answer path rather than maximizing topK by intuition.

4. Security and data isolation

Permission-aware retrieval needs to be enforced outside the model. Test cross-tenant access, role changes, deleted permissions, cached results, and citation links.

5. Evaluation

The original notes use three useful questions:

Context relevance: Did retrieval find evidence related to the question?
Groundedness or faithfulness: Does the answer stay within the supplied evidence?
Answer relevance: Does it answer what the user asked?

Add operational checks that match the system, such as extraction failures, stale indexes, permission denials, and citation integrity. Do not invent target scores. Establish them from your risk and user requirements.

The main engineering lesson

RAG connects language-model capability with external knowledge, but the connection is only as useful as the pipeline around it.

A local demo can hide document quality, authorization, stale versions, and evaluation because the sample chunks are already clean. Production exposes all of them.

For me, the work starts before prompt engineering:

prepare and version the knowledge
choose chunk boundaries that preserve meaning
retrieve with semantic and exact-match needs in mind
apply permissions before context construction
preserve citations
evaluate retrieval and generation separately

The LLM writes the final response. Most of the evidence it can use has already been decided by then.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

How to Write an AI Agent Skill File That Reduces Guesswork

NEXT4I DEV — Mon, 14 Sep 2026 16:58:08 +0000

A prompt can tell an agent what to do once. A skill file can document how a team handles that class of work repeatedly.

For developers, the distinction matters. An agent may have repository access and the right tools, yet still has to make decisions about scope, references, validation, and failure handling. If those decisions are undefined, a technically valid result can still be the wrong result.

This guide builds a small, generic skill for reviewing API changes. The point is not to prescribe one universal Agent Skills format. It is to show how to turn vague intent into an executable workflow.

Skill discovery, instruction loading, and tool use vary by model, agent runtime, system instructions, and available tools. Treat the structure below as a portable design pattern, then adapt it to your platform.

The problem with a reasonable instruction

Consider this prompt:

Review this API and make sure it follows best practices.

It sounds clear, but the agent still has to decide:

Which API contract is authoritative?
Does the review include authentication and permissions?
Should it modify code or only produce a report?
Which tests should it run?
What should it do when information is missing?
What does a completed review contain?

Different models may fill those gaps differently. The result can be reasonable without matching the workflow you intended.

Prompt versus skill

I use this distinction:

A prompt says, “Do this task now.”
A skill says, “This is how we handle this type of task.”

A useful skill does not make a model magically smarter. It makes the operating boundaries visible.

At minimum, it should define:

Trigger: when the skill applies
Goal: what outcome it should produce
Scope: what is included and excluded
Workflow: steps the agent can follow
References: information to read under specific conditions
Constraints: actions and claims that are prohibited
Uncertainty handling: what to do when evidence is missing
Completion criteria: how to validate the result

A practical directory structure

Start with one file while the workflow is small. Split it when references, examples, or reusable scripts become large enough to distract from the main instructions.

api-review-skill/
├── SKILL.md
├── references/
│   ├── api-contract.md
│   ├── security-rules.md
│   ├── output-examples.md
│   └── troubleshooting.md
├── scripts/
│   └── validate.sh
└── assets/
    └── review-template.md

The exact file names and supported directories depend on the platform and runtime. More importantly, models may not select or load these files in the same way. The main file should therefore explain why and when each reference matters.

A generic `SKILL.md` example

---
name: api-change-review
description: "Review API changes for contract compatibility,"
  security boundaries, and required validation. Use when an
  endpoint, request schema, response schema, authentication,
  or permission behavior changes.
---

# API Change Review

## Goal

Produce a review report that identifies contract-breaking changes,
security-sensitive changes, missing validation, and unresolved assumptions.

## Scope

Review the proposed change and related tests.
Do not modify implementation files unless the user requests edits.

## Workflow

1. Read the changed files and the API contract.
2. Identify changes to endpoints, fields, status codes, and behavior.
3. If authentication, secrets, or permissions are involved,
   read the security rules.
4. Compare the tests with the changed behavior.
5. Produce the required report.
6. Run the validation command when the environment supports it.

## Reference routing

- Read `references/api-contract.md` for endpoint or schema changes.
- Read `references/security-rules.md` for authentication,
  secrets, or permission changes.
- Read `references/output-examples.md` only when the report format
  is unclear.
- Read `references/troubleshooting.md` when validation fails.

## Constraints

- Do not invent endpoints, fields, metrics, or test results.
- Do not expose credentials, customer data, or internal URLs.
- Do not present an unimplemented requirement as released behavior.
- Flag assumptions that require human review.

## Required output

1. Summary
2. Contract-breaking changes
3. Security-sensitive changes
4. Missing or affected tests
5. Assumptions requiring review
6. Validation performed

## Completion criteria

- Every changed behavior is mapped to the contract.
- Security-sensitive changes are explicitly identified.
- Validation results state what was and was not run.
- Unresolved assumptions are visible.

This example is intentionally generic. It demonstrates an instruction contract, not a claim about one platform's required syntax.

Before and after

Before

Review this API and make sure it follows best practices.

After

Review the API change using the api-change-review skill.

Use `references/api-contract.md` as the contract source.
If the change touches authentication, secrets, or permissions,
also apply `references/security-rules.md`.

Produce the required review report. Do not modify implementation files.
State which validation commands were run and flag unresolved assumptions.

The second instruction is not better because it is longer. It is better because fewer operational decisions are left undefined.

Turn the index into a routing table

A file list tells an agent what exists. A routing table explains when a file becomes relevant.

Reference	Read when	Skip when
`api-contract.md`	An endpoint, schema, status code, or behavior changes	The task does not involve an API contract
`security-rules.md`	Authentication, secrets, roles, or permissions are involved	The task has no security-sensitive behavior
`output-examples.md`	The required report shape is unclear	The output contract is already explicit
`troubleshooting.md`	Validation cannot run or fails	Validation succeeds normally

This table is guidance, not an enforcement mechanism. Actual reference selection still depends on the model, runtime, context strategy, and tools.

Single file or multiple files?

Keep the skill in one file when:

it has one short workflow
all constraints fit without hiding the main steps
examples are small
no reusable scripts are needed

Split the skill when:

references are long or domain-specific
several workflows share the same rules
examples make the main file difficult to scan
validation scripts are reusable
sensitive rules require separate ownership or review

The main file should remain enough to answer two questions: What should the agent do, and where should it look next?

Treat skills like software

A skill can contain instructions, references, and executable code. Review an external skill before using it, especially when it can access files, call a network service, or run scripts.

Check for:

file access outside the expected scope
network and API destinations
hardcoded credentials
destructive file operations
hidden instructions that attempt to bypass system rules
scripts whose behavior has not been reviewed

For organizational use, sandboxing and coexistence tests may also be appropriate. Different models can respond to the same instruction or tool surface differently.

Validation checklist

- [ ] The description states when the skill should trigger.
- [ ] The goal and scope are explicit.
- [ ] The workflow contains observable steps.
- [ ] Each reference has a condition for when to read it.
- [ ] Prohibited actions and claims are listed.
- [ ] Missing information has a defined handling rule.
- [ ] The output contract is reusable and testable.
- [ ] Completion criteria can be checked.
- [ ] No secrets or internal URLs are embedded.
- [ ] Executable scripts have been reviewed.
- [ ] The skill was tested on tasks that should trigger it.
- [ ] The skill was tested on tasks that should not trigger it.

Final takeaway

Before handing a skill to an agent, read it as if you were a developer joining the project today:

Can I complete the work from this information? What would I still have to guess? If I must make a decision, do I know which direction to take?

A good skill does not remove reasoning. It removes avoidable guessing.

Read the canonical article on NEXT4I: https://go.next4i.com/next4i-devto-en

What Is Token & LLM Cost Optimization

NEXT4I DEV — Mon, 07 Sep 2026 14:35:04 +0000

What Is a Token in LLMs? A Developer's Guide to Cost Optimization and Architecture

Understanding how LLM tokens work is the difference between an AI feature that costs $50/month and one that runs up a $5,000 AWS/OpenAI bill.

Here is the engineering breakdown of tokenization algorithms, input vs. output pricing asymmetries, multilingual overhead, model pricing dynamics, and six production strategies implemented at NEXT4I to cut inference costs by up to 50-80%.

#Buildinpublic, #ModelAI, #LLM, #AIArchitecture, #TokenOptimization, #AIDeveloper

1. What Is a Token Under the Hood?

LLMs do not process raw text or strings. They consume Tokens—subword representations mapped to high-dimensional embedding vectors via algorithms like Byte-Pair Encoding (BPE).

"Hello world" -> ["Hello", " world"] (2 tokens)
"Unstoppable" -> ["Un", "stoppable"] (2 tokens)

The Multilingual Token Penalty

Because vocabulary dictionaries are predominantly trained on English corpora, languages without whitespace delimitation (such as Thai) suffer severe subword fragmentation:

English: 1 Token ≈ 0.75 words (~4 characters).
Thai: 1 Word ≈ 3 to 8 Tokens (frequently split into byte-level representations).

A prompt written in Thai can cost up to 6x more in raw token usage and introduce noticeable latency compared to its English equivalent.

2. Formatting Mechanics: Linebreaks & The "Emoji Tax"

Every character in your prompt payload carries a token cost:

Newlines (\n) and Numbered Markdown:
- Consumes 1 token per newline.
- Verdict: Highly recommended. Clear formatting provides structural anchors for transformer attention heads, significantly reducing hallucination.
The Emoji Tax:
- Emojis are complex multibyte Unicode sequences, often consuming 2 to 6+ tokens each.
- Avoid embedding decorative emojis in static system prompts that execute millions of times.

3. Input vs. Output Tokens: Why Output Costs 3x–5x More

API rate cards price Output Tokens significantly higher than Input Tokens. Why?

Input Tokens (Parallelized Compute): Processed simultaneously across GPU tensor cores in a single matrix multiplication pass (Prefill / Encoding). It is fast and hardware-efficient.
Output Tokens (Sequential Autoregression): Generated one token at a time (Autoregressive Decoding). The model generates token N, appends it back to the context history, and re-computes attention for token N+1. This locks GPU resources over the entire generation cycle.

4. Why Model Pricing Varies by 100x (Dense vs. MoE Architecture)

You may have noticed that API pricing spans from $0.30 to $50.00+ per 1 million tokens across models:

DeepSeek-V4: $0.50 – $1.70 / 1M Tokens (Massive architecture, exceptionally cost-efficient)
Gemini 3.7 Flash: $0.30 – $1.80 / 1M Tokens (High-efficiency edge tier)
Claude Sonnet 5: $2.00 – $10.00 / 1M Tokens (Mid-tier balanced powerhouse)
OpenAI GPT-5.6 Terra: $2.00 – $12.00 / 1M Tokens (Enterprise mid-tier)
Claude Opus 5: $5.00 – $25.00 / 1M Tokens (Premium reasoning class)
Claude Fable 5: $10.00 – $50.00 / 1M Tokens (Mythos-class model from Anthropic)
OpenAI GPT-5.6 Sol: $4.00 – $30.00 / 1M Tokens (OpenAI's flagship)

Note: Model pricing per token fluctuates frequently across providers and should be used strictly for relative comparative analysis.

This pricing variance stems from three fundamental drivers:

1. Architectural Design: Dense Models vs. Mixture-of-Experts (MoE)

Dense Models

Mechanics: Every incoming vector/token is processed through 100% of the model's parameters, from the initial layer to the final output layer.
Analogy: Like a company where every single employee must sit in every meeting and vote on every decision—regardless of whether the task is simple arithmetic or legal compliance.
Trade-offs: Highly compute-intensive (massive FLOPs per token), resulting in higher per-token inference costs. However, memory management and GPU scheduling remain straightforward.

Mixture-of-Experts (MoE)

Mechanics: An intelligent gating mechanism called a Router (or Gating Network) acts as a dispatcher, evaluating incoming tokens and dynamically routing them to specialized sub-networks (Experts).
Execution Flow:
1. The Router analyzes each token and activates only a sparse subset of experts (e.g., selecting 2 out of 8 or 64 total experts).
2. Compute flows exclusively through the parameters of the chosen experts.
Analogy: Like a well-structured organization with an executive dispatcher. A calculus question is routed strictly to the math experts without distracting the linguistics team.
Advantages: Enables total model capacity (Total Parameters) to scale massively while keeping active compute per token (Active Parameters / FLOPs) extremely low. This allows lightning-fast generation and drastically lower API prices (e.g., DeepSeek, Mixtral).
Trade-offs: Requires massive VRAM/RAM pools to keep all expert weights loaded in memory simultaneously.

The MoE Achilles' Heel: When Routers Fail

While MoE unlocks unmatched cost efficiency, its performance is tightly bound to routing stability:

Loss of Nuance & Context Disruption:
- Natural language is rich with subtle subtext. If a router misinterprets a token and dispatches it to the wrong expert, nuanced meaning collapses. The output may stay grammatically intact but lose analytical depth or fail to address the core prompt intent.
Router Collapse & Expert Imbalance:
- Routers naturally develop bias toward a handful of frequently trained experts, causing severe load imbalance. The favored experts hit computational bottlenecks while neglected experts become Dead Parameters, defeating the entire purpose of modular specialization.
Cascading Errors:
- LLMs process representations layer by layer. If an early-layer router misroutes a token, downstream layers receive corrupted intermediate activations, amplifying routing errors across subsequent layers.

🛠️ Engineering Safeguards Used by Frontier Labs:

Auxiliary Load Balancing Loss: Incorporating penalty penalties into the loss function during pre-training to enforce uniform token distribution across all expert sub-networks.
Capacity Factor Enforcement: Setting strict token buffer caps per expert. Once an expert's capacity threshold is reached, excess tokens are spilled over to secondary experts to prevent execution bottlenecks.

2. Reasoning Overhead (Thinking / Chain-of-Thought Tokens)

Reasoning-focused models (like Claude Opus, OpenAI GPT Sol) generate thousands of internal, hidden Chain-of-Thought tokens before emitting their first visible output token. Providers meter and bill for every single background reasoning step.

3. Hardware Sovereignty & Custom Silicon

Hyperscalers operating proprietary custom silicon (such as Google’s TPU clusters for Gemini) achieve significantly lower baseline operating costs than providers renting general-purpose Nvidia H100/H200 GPU clusters.

5. 6 Production Strategies to Cut Token Costs by 80%

Here are some of the production-level strategies we use in building NEXT4I:

1. Prompt Engineering for Token Efficiency

Eliminate fluff and instructions that don't add semantic value. Use concise formats like YAML or Markdown instead of verbose JSON schemas.

2. Leverage Prompt Caching

Major LLM providers (Anthropic, OpenAI, DeepSeek, Google) offer prompt caching. Placing static context (system instructions, tool definitions, schemas) at the prompt root allows providers to cache the KV-cache, reducing input costs by 75%–90%.

3. Intelligent Model Cascading (Model Routing)

Never route every query to flagship models. Route simple classification, data extraction, and formatting tasks to lightweight models or budget-friendly models (Gemini Flash, DeepSeek), escalating only complex reasoning tasks to larger models (Claude Sonnet, GPT Terra, GPT Sol, Claude Opus / Fable).

4. Sliding Context Windows & Conversation Pruning

Chat histories grow quadratically ($O(n^2)$) if sent in their entirety on every turn. Maintain a rolling sliding window of the last 5–10 messages, or summarize older context into a single concise paragraph.

5. Pre-Retrieval RAG Filtering

In RAG pipelines, do not inject full documents into the context window. Use embedding similarity and rerankers to select top-k (3 to 5) chunks, applying semantic deduplication before prompt construction.

6. Multilingual Translation Layer

For bulk data extraction or batch processing on non-Latin languages, translating text to English with a lightweight model prior to deep inference on flagship models can reduce total token usage and improve execution latency.

Key Takeaways

+-------------------+---------------------------------------------------+
| Metric            | Engineering Reality                               |
+-------------------+---------------------------------------------------+
| Token Ratio (EN)  | ~1 Token ≈ 0.75 words (~4 characters)             |
| Token Ratio (TH)  | ~1 Word ≈ 3–8 Tokens (byte-level inflation)       |
| Cost Ratio        | Output is 3x–5x more expensive than Input         |
| Pricing Deltas    | Dense vs MoE, TPU/ASIC custom silicon, CoT tokens |
| Core Optimizers   | Caching + Routing + Windowing + RAG Filtering     |
+-------------------+---------------------------------------------------+

Treating tokens as finite compute bandwidth ensures your AI infrastructure remains fast, scalable, and economically sustainable.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

The RAG War Story: How I Learned to Stop Worrying and Love Markdown

NEXT4I DEV — Tue, 01 Sep 2026 11:23:36 +0000

TLDR; When building an AI knowledge retrieval pipeline that extracts text from documents, I discovered that PDF is the worst format for AI and Markdown is the best. Here's the multi-step pipeline I had to build just to handle PDFs, why it was necessary, and the generic pattern you can steal to handle document ingestion in your own RAG systems.

The Problem: PDFs Are Pixel-Perfect Hell for AI Parsers

I was building a RAG (Retrieval-Augmented Generation) pipeline for NEXT4I the kind of system that reads your documents first, then answers questions from them. Standard stuff: document ingestion → chunking → embedding → vector search → LLM answer generation.

I chose a beautiful Thai tourism PDF as my test document. Professional design, complex Thai typography, images, tables, charts the works. Real-world document, real-world pain.

Here's what the naive approach looked like:

PDF File → PDF Parser → Extracted Text → Chunk → Embed → Search

And here's what actually worked:

PDF File
  ├─→ PDF Parser → Raw Text (broken Thai, missing punctuation)
  ├─→ Page Renderer → Full-Color Images
  │     └─→ B&W Converter → High-Contrast Images
  ├─→ AI Vision Model (color images) → Image Descriptions
  ├─→ AI Vision Model (B&W images) → Text Extraction
  └─→ Cross-Validation Layer
        ├─→ Multi-Model Synthesis
        ├─→ Spell-Check Model (critical for Thai)
        └─→ Human Review
              └─→ Final Structured Text → Chunk → Embed → Search

Why the complexity? Because PDF is fundamentally a presentation format, not a data format. When you extract text from a PDF, you're not reading structured data you're reverse-engineering a rendered page layout. For languages with complex typography like Thai (where vowels can appear above, below, left, or right of consonants, and tone marks float above), this is especially brutal.

The Generic Pattern: Multi-Path Document Ingestion with Cross-Validation

If you're building any system that ingests arbitrary documents, you'll inevitably hit the PDF wall. Here's the reusable pattern I settled on:

Architecture

                    ┌──────────────┐
                    │   Document   │
                    │   Ingest     │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Direct  │ │  Image   │ │  Image   │
        │  Text    │ │  (Color) │ │  (B&W)   │
        │ Extract  │ │  Render  │ │  Render  │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │  Text    │ │  Vision  │ │  Vision  │
        │  Output  │ │  Model   │ │  Model   │
        │          │ │  (Desc)  │ │  (OCR)   │
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │            │            │
             └────────────┼────────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Cross-Validate │
                 │  & Synthesize   │
                 │  (Multi-Model)  │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Spell-Check    │
                 │  & Normalize    │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Human Review   │
                 │  (Optional)     │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Structured     │
                 │  Output → Embed │
                 └─────────────────┘

The key insight: no single extraction path is reliable enough on its own. You need multiple independent paths producing results, then a synthesis layer that cross-validates. Think of it like sensor fusion each path is a noisy sensor, and the truth emerges from the overlap.

Why Spell-Check is Non-Negotiable for Non-English Languages

For English, you might get away without a dedicated spell-check pass. For Thai where a single misplaced tone mark changes the entire word you absolutely cannot. OCR and vision models hallucinate characters constantly on decorated fonts or text-over-image backgrounds. A dedicated language model fine-tuned for spell correction is the difference between "usable" and "garbage."

The Real Takeaway: Markdown is AI-Native. Everything Else Is Legacy.

After building this entire pipeline, I had a moment of clarity. If that same document had been authored in Markdown:

## Top Destinations

| Province | Highlight | Best Season |
|----------|-----------|-------------|
| Krabi    | Islands   | Nov–Apr     |
| Chiang Mai | Mountains | Nov–Feb   |

See the [full itinerary](#itinerary) for details.

mermaid
graph TD
A[Arrive Bangkok] --> B[Fly to Krabi]
B --> C[Island Hopping]
C --> D[Return]

markdown

...the entire pipeline collapses to: read the file → chunk → embed → search. That's it.

No OCR. No vision models. No B&W conversion. No multi-path cross-validation. No spell-check model. No human review for format-induced errors.

Markdown is structured, plain-text, and both human-readable and machine-parseable by default. It's the only format where:

Headings are unambiguously # / ## not inferred from font size
Tables are | column | row | syntax not pixel grids
Diagrams are Mermaid text not flattened raster images
Code is fenced not monospaced-font heuristics

"And here is how the human user experiences it:"

Top Destinations

Province	Highlight	Best Season
Krabi	Islands	Nov–Apr
Chiang Mai	Mountains	Nov–Feb

See the full itinerary for details.

graph TD
    A[Arrive Bangkok] --> B[Fly to Krabi]
    B --> C[Island Hopping]
    C --> D[Return]

"In reality, we can't always control the documents we ingest, and we can't just ignore them because they might contain critical data. But if we were to start from scratch, Markdown is definitely the go-to choice."

How This Shapes Our Architecture at NEXT4I

At NEXT4I, we treat Markdown as a first-class format throughout our stack. When building AI knowledge retrieval systems for everyday users and organizations, we encourage Markdown as the source of truth and handle PDFs as a necessary-but-painful compatibility layer.

The design principle is simple: AI Integration by Design. Make AI a first-class citizen of your content architecture, not something you bolt on later and hope it works. The format you choose today determines the ceiling of your AI capabilities tomorrow.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

What Is LLM Actually Doing? A Fellow Engineer's Take on Vectors, Next-Token Prediction, and Fail-back Routing

NEXT4I DEV — Tue, 01 Sep 2026 05:30:51 +0000

Why treating an LLM as a probability engine, not a brain, changes how you architect around it. The reasoning behind NEXT4I's AI layer.

#LLM #BuildinPublic #SystemArchitecture #AI #Model AI #AI Router #AI Stable

I used to wonder why the model confidently gives you a wrong number. It's not a bug in the traditional sense, it's the model doing exactly what it's built to do: predicting the next token from probability, with zero actual arithmetic happening underneath.

TLDR; An LLM (large language model) converts text into vectors (numeric coordinates in a high-dimensional meaning-space) and generates output via next-token prediction, sampled with parameters like top-k and temperature. Because it's fundamentally a probability engine and not a calculator or a database, I designed NEXT4I with automatic model fail-back routing and task-based model selection instead of trusting any single model as a source of truth.

The Mental Model: Vectors, Not Meaning

Every token gets embedded into a vector, often with hundreds or thousands of dimensions. Semantically similar tokens end up close together in that space. That's why semantic search (vector-based retrieval) can match "large flying animal consumes insects" to "big bird eats worms" even with zero shared keywords, unlike old-school lexical search (TF-IDF/BM25) which needs literal term overlap.

GPUs handle this well because they're already wired for massive parallel floating-point math (the same math used to shade millions of pixels per frame), so throwing billions of similarly-directed vectors at a GPU is a natural fit, not a coincidence.

Generation Is Sampling, Not Retrieval

Given a prompt, the model doesn't look up an answer, it samples one token at a time from a probability distribution. Two knobs matter in practice:

top_k: 2        # only sample from the top-2 most likely next tokens
temperature: 0.2  # low = deterministic/precise, high = creative/varied

Low temperature + low top_k gives you consistent, "boring" output, good for structured extraction. High temperature gives you variety, good for brainstorming, bad for anything requiring precision.

Why LLMs Hallucinate on Arithmetic

There's no calculator inside the model. It doesn't evaluate x * y, it predicts digits that are statistically plausible given the prompt, one token at a time. It gets 2 * 2 right because that pattern is everywhere in training data. It confidently botches large multiplication because it's still just sampling digits, not computing. This is exactly why production systems now delegate real math to a tool call (a Python sandbox, a calculator function) instead of trusting raw model output.

Core Value: A Generic Model-Tier Fail-back Router

Here's the simple pattern, stripped of any specific business logic, a reusable fail-back wrapper for any set of same-tier model clients:

package modelrouter

import (
    "context"
    "errors"
    "fmt"
)

// ModelClient is any backend that can answer a prompt.
type ModelClient interface {
    Name() string
    Complete(ctx context.Context, prompt string) (string, error)
}

// TieredRouter tries each client in a tier in order until one succeeds.
type TieredRouter struct {
    tier []ModelClient
}

func NewTieredRouter(clients ...ModelClient) *TieredRouter {
    return &TieredRouter{tier: clients}
}

// Complete attempts each model in the tier, fail-back on error.
func (r *TieredRouter) Complete(ctx context.Context, prompt string) (string, error) {
    var errs []error
    for _, client := range r.tier {
        resp, err := client.Complete(ctx, prompt)
        if err == nil {
            return resp, nil
        }
        errs = append(errs, fmt.Errorf("%s: %w", client.Name(), err))
    }
    return "", errors.Join(errs...)
}

This is intentionally boring: try the next model in the same tier on failure, return the first success. No retries with backoff yet, no circuit breaker, just the core fail-back idea. In NEXT4I's actual implementation, tiers are populated dynamically and health state feeds back into ordering, but that logic is abstracted here on purpose, the generic version above is what's actually useful to share.

Core Value: Task-Based Routing, the Simple Version

A minimal router that inspects task complexity before picking a tier:

package modelrouter

type Complexity int

const (
    Simple Complexity = iota
    Complex
)

func ClassifyAndRoute(task string, simpleTier, complexTier *TieredRouter) *TieredRouter {
    if estimateComplexity(task) == Simple {
        return simpleTier
    }
    return complexTier
}

func estimateComplexity(task string) Complexity {
    if len(task) < 100 {
        return Simple
    }
    return Complex
}

estimateComplexity can be as crude as a length/keyword heuristic or as sophisticated as a small classifier model, the point is the routing decision happens before the expensive call, not after.

Trade-offs I Made

Fail-back within a tier, not across tiers. Swapping a cheap model in for an expensive one silently would change output quality without anyone noticing. Tiers exist specifically to avoid that.
No cross-request state in the router. Keeps it stateless and trivially horizontally scalable, at the cost of not learning from past failures within a single request lifecycle.
Routing is a toggle, not a mandate. Users can pin a specific model when they need deterministic behavior from one exact provider, the router only kicks in by default.

If there's one thing worth taking away: treat the model as a probability engine you can't fully trust, and let the system around it, fail-back, routing, tool calls for math, carry the reliability burden instead.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

How to Build a "Second Brain" with Obsidian That Your AI Agent Can Read, Without Building a Custom RAG Pipeline

NEXT4I DEV — Mon, 17 Aug 2026 11:40:46 +0000

Ever run into this? You wrote a detailed technical spec three months ago, and today someone asks "how did we design this module again?" You end up spending 20 minutes hitting Cmd+F across Google Docs, Trello, and README.md files scattered across different repos, or sometimes there's no documentation at all, so you have to dig through the code and reverse-engineer it by eye.

TLDR; I built a knowledge-base system for NEXT4I using Obsidian (plain Markdown) together with Git and a VS Code AI agent, without writing a single line of custom integration. The core of the system is choosing tools that already "speak the same language" from day one (plain text, open formats), so AI can read our knowledge base practically for free, no RAG pipeline required.

Pain Point: Scattered Knowledge and Requirements Make Search Hard and Nothing Stays in Sync

Before this system, all of NEXT4I's knowledge was scattered across many places: notebooks, Apple Notes, Google Docs, Google Sheets, Trello, README.md files, or, even worse, sometimes there was no documentation at all, just buried in the code, spread across multiple repos each written in a different language, frontend and backend alike.

This isn't just an inconvenience. It makes search genuinely hard, sometimes it's Cmd+F and pray. Worse: my AI coding agent could read the entire codebase, but it had zero visibility into the reasoning behind that code, because that reasoning was scattered somewhere the AI couldn't reach.

Design Constraints: 3 Non-Negotiables

Before picking a tool, I set 3 rules:

It has to be affordable, or free if possible.
It has to be accessible online anytime, from my phone, and still work offline.
Plain text, zero lock-in. If the tool disappears tomorrow, the files must remain immediately readable and usable.
Local-first, Git-friendly. It has to be a normal folder I can git init and track right away.
AI-readable without building extra infrastructure. My AI agent already lives in VS Code, so the knowledge base has to sit inside that workspace without me building an extra pipeline.

Obsidian passed everything, because at its core, an Obsidian vault is just a folder of .md files.

Architecture: Vault Structure

vault/
├── Ideas/          # Raw concepts, brainstorming
├── Manifesto/      # Vision, mission, core policies
├── Principles/     # Design rules, engineering guidelines
├── Infrastructure/ # Deployment topology, IaC specs
├── Platform/       # Domain model, API contracts
├── Script/         # Utility scripts, automation, runbooks
├── Skill/          # Patterns, checklists, reusable knowledge
└── Appendix/       # Domain glossary, citations

Every file is a plain .md. Links use [[wiki-link]] syntax. Metadata lives in YAML frontmatter, and Graph View renders the relationships as a visible dependency graph.

Version control is just git init inside the vault folder, then committing every change with a rationale. git log -- "Infrastructure/sharding-strategy.md" shows the full decision history for that topic. Push it to a private GitHub repo and you get backup, an audit trail, and branching for major revisions, all for free.

Key Insight: AI Access Without Building Anything

This is the part that changed everything.

Obsidian vault = a folder of .md files.
VS Code = opens any folder as a workspace.
AI coding agent (running as a VS Code extension) = reads every file in that workspace.

So the "integration" here is: open the vault folder in VS Code.

That's it. No API, no embedding pipeline, no vector database, no chunking strategy. Just plain Markdown files that the AI agent reads natively.

Prompt Patterns I Actually Use

Contextual search + reasoning:

Search all documents that mention our sharding strategy.
Summarize every trade-off we've considered
and tell me which approach we ultimately chose and why.

Gap analysis:

Look at everything in the Infrastructure/ folder
and tell me which architectural decisions are still undocumented,
compared against the template in Skill/.

Drafting from conventions, not from a blank page:

Using the patterns in Skill/go-backend/ and the domain model in Platform/core/,
draft a design doc for a new message consumer
following our established conventions.

Impact analysis via link traversal:

If I change the authentication rule in Principles/auth.md,
trace every file in Platform/ and Skill/ that links to it via [[links]]
and tell me what needs updating.

The AI reads across multiple files, follows [[wiki-links]], understands the relationships, and synthesizes an answer, without me writing a single line of integration code.

Philosophy: Seamless Integration by Design

The pattern here isn't "integrate 3 tools." It's choosing components that already speak the same language.

Obsidian chose Markdown, the most universally readable format in computing, over a proprietary database. Git works with any plain text. AI agents already know how to read files in a VS Code workspace.

Bonus: Obsidian Canvas as a Visual Layer

Obsidian's Canvas is an infinite whiteboard, place document cards, text, media, then draw connections between them.

I use Canvas for:

System architecture sketches: each service as a card, data flow arrows, real specs embedded right on the board
Decision trees: "if we pick X, then Y and Z are affected," with linked evidence
Plan strategy & flow: for planning work, sequencing, and various NEXT4I workflows

Canvas files are Markdown too under the hood (JSON-like structure), so they're Git-versioned and AI-readable as well. Drop a .canvas file into your VS Code workspace and ask the AI to analyze it for circular dependencies or single points of failure.

What I Learned

What makes this system work isn't the technology, it's what I didn't build. No middleware, no proprietary pipeline, no vendor lock-in.

The discipline of plain text + Git + open formats is a feature, not a limitation.

If you're a dev or a small team drowning in scattered documents, try this before jumping to a heavyweight knowledge-management platform. Plain Markdown with a good folder structure will take you further than you'd expect.

Thanks for reading all the way to the end, I'll keep working on more articles like this.

Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i-devto-en

Alternatively, you can register to join NEXT4I the AI-Native Ecosystem I am currently building at: https://go.next4i.com/cta

tags:
#buildinpublic
#secondbrain
#next4i
#obsidian

How I Built an AI-Readable Second Brain with Obsidian, Git, and a VS Code AI Agent

NEXT4I DEV — Fri, 07 Aug 2026 07:55:45 +0000

When you're a solo founder and lead architect, your knowledge base is your most valuable asset. Lose the thread on why a decision was made, and you spend hours — sometimes days — reconstructing context that you already figured out once.

I want to share the exact setup I use at NEXT4I to turn a folder of Markdown files into a fully searchable, version-controlled, AI-readable knowledge system. No proprietary SaaS, no vendor lock-in, no custom integration work.

This is the Key Highlight of this post: a genuinely useful, generic pattern you can apply to your own projects today. The NEXT4I-specific business logic stays abstracted (per our security rules), but the pattern itself is 100% reusable.

1. The Architecture: Three Layers, Zero Magic

┌─────────────────────────────────────────┐
│           AI Agent (VS Code)            │
│   Reads, Searches, Summarizes, Drafts   │
└──────────────────┬──────────────────────┘
                   │ reads plain .md files
┌──────────────────▼──────────────────────┐
│       Git-tracked Obsidian Vault        │
│  ├── Idea/           (brainstorms)      │
│  ├── Infrastructure/ (architecture docs)│
│  ├── Platform/       (product specs)    │
│  ├── Script/         (automation)       │
│  └── Skill/          (reusable limits)  │
└──────────────────┬──────────────────────┘
                   │ committed & pushed
┌──────────────────▼──────────────────────┐
│         GitHub (remote backup)          │
└─────────────────────────────────────────┘

Layer 1 — The Vault (Obsidian): A folder of interconnected .md files. The key insight is that Obsidian uses plain Markdown with [[wiki-links]] for connections — no database, no proprietary format.

Layer 2 — Version Control (Git): Every vault is a git repo. Every change to any document has a commit message, a timestamp, and a diff. You can git log --oneline -- Idea/ to see the evolution of a concept.

Layer 3 — AI Agent (VS Code Extension): Because the vault is just a file tree of .md files, any AI coding agent that can read a codebase can also read your knowledge base. Point the agent at the vault folder, and it has full context.

2. The Setup: Step-by-Step

Step 1: Create the Vault


mkdir next4i-knowledge

cd next4i-knowledge

mkdir Idea Infrastructure Platform Script Skill

git init

Open this folder in Obsidian: Open folder as vault.

Step 2: Link Everything

Inside a note, link to another note with [[Note Name]]. Obsidian auto-suggests as you type. Over time, this builds a graph you can visualize with Cmd/Ctrl + G.

Pro tip: Create a _INDEX.md in each folder that links to the most important notes. This becomes a human-readable table of contents AND a search anchor for the AI.

Step 3: Add Git Discipline


git add -A && git commit -m "infra: initial sharding strategy decision"

git remote add origin git@github.com:your-org/knowledge-vault.git

git push -u origin main

Treat commit messages like code. Use prefixes: idea:, infra:, platform:, script:, skill:. This makes git log --oneline --grep="infra:" instantly useful.

Step 4: Open in VS Code and Activate the AI


code /path/to/vault

With an AI agent extension active (Copilot, Cline, Cody, etc.), try prompts like:

"Summarize the key architectural decisions in the Infrastructure folder."
"Find any contradiction between documents in /Platform/ and /Infrastructure/."
"Draft a new document in /Idea/ based on the sharding notes in /Infrastructure/."

The agent reads the files as context, just like it would for code.

3. The Design Pattern: Folder-Convention-as-API

Here's the key pattern: your folder structure IS your API.

By keeping a consistent vault structure, both humans and AI know where to look:

Folder	Contains	AI Use Case
`Idea/`	Raw, unstructured thinking	Generate summaries, find related concepts
`Infrastructure/`	System topology, deployment, config	Validate consistency, trace dependencies
`Platform/`	Feature specs, user flows	Draft task tickets, check requirement coverage
`Script/`	Automation, one-liners	Explain what a script does, suggest improvements
`Skill/`	Reusable patterns, checklists	Retrieve relevant patterns for new tasks

This is essentially a convention-based RAG (Retrieval-Augmented Generation) setup without any vector database, embedding pipeline, or chunking strategy. The "chunking" is the natural boundary of each .md file. The "retrieval" is the AI agent's file-reading capability.

4. Why This Beats a Wiki

Wiki / Confluence	This Setup (Obsidian + Git)
Vendor lock-in	Plain `.md` files, portable anywhere
Search is siloed within the tool	VS Code AI searches across the whole vault
No version control (or poor built-in)	Full version control (`git blame`, `git diff`, `git log`)
Hard to automate	Scriptable — `grep`, `sed`, and AI prompts all work
AI needs API integration	AI reads files natively, zero setup required

5. What I Learned

The biggest surprise: The AI agent became better at finding connections in my own notes than I was. It doesn't have recency bias. It doesn't forget what I wrote 8 months ago. It reads everything with equal attention.

The biggest lesson: AI-native doesn't mean "add an AI button." It means design your systems — including your thinking systems — so that AI can participate as a first-class citizen without special plumbing.

I'm building NEXT4I as an AI-native ecosystem from the ground up. If you're interested in following a solo founder's engineering journey — or want early access — join here:
Subscribe NEXT4I or want early access — join here

DEV Community: NEXT4I DEV

Chunking in RAG: How to Split Documents Without Losing the Conditions

TL;DR

What is chunking, and why not use the entire file?

One boundary can change the answer

The main approaches and their trade-offs

1. Fixed-size splitting

2. Fixed-size splitting with overlap

3. Recursive splitting

4. Structure-aware splitting

5. Semantic chunking

6. Parent-child retrieval and contextual chunking

Match the approach to the document

The Thai tourism PDF that changed my priorities

How large should chunks and overlap be?

How to tell whether chunking works

The goal is usable evidence

From Ctrl + F to AI-Powered Search

How Do Computers Find Content and Answer Questions from Files?

1. The Fundamental Gap: Humans Read Meaning, Computers Read Bytes

2. Era 1: Exact Character Matching (Lexical / Exact Keyword Search)

2.1 Linear Scan (Ctrl + F or grep)

2.2 Inverted Index (The Backbone of Search Engines)

Limitations of Lexical Search: Why Exact Matching Falls Short

3. Era 2: Searching by Meaning (Semantic / Vector Search)

3.1 What is Text Embedding? (Turning Language into Coordinates)

3.2 Measuring Proximity Mathematically (Similarity Metrics)

4. Why Not Convert an Entire 100-Page File into a Single Vector?

The Solution: The Art of Chunking (Chunking Strategy)

5. From Finding Content to Answering Questions: The 3-Step Pipeline

6. Comparison: 4 Paradigms of Document Retrieval

7. Conclusion: What Should Real-World Systems Use?

RAG Architecture Beyond the Demo: Retrieval, Thai Chunking, and Production Boundaries

The three limits RAG is trying to address

Knowledge boundaries

Context and cost boundaries

Verifiability

RAG is two pipelines, not one request

Ingestion quality becomes retrieval quality

Thai documents make naive chunking easier to break

Ambiguous boundaries

Long dependencies

Periods that are not sentence endings

Chunking strategies and their trade-offs

Fixed size with overlap

Recursive splitting

Semantic chunking

Parent-child chunking

Embeddings are coordinates, not facts

Conceptual implementation in Python (In-Memory Retrieval & Query Pipeline)

Dense search is not enough for every query

Five production boundaries worth testing

1. Language and embedding fit

2. Extraction and version quality

3. Context pollution

4. Security and data isolation

5. Evaluation

The main engineering lesson

How to Write an AI Agent Skill File That Reduces Guesswork

The problem with a reasonable instruction

Prompt versus skill

A practical directory structure

A generic SKILL.md example

Before and after

Before

After

Turn the index into a routing table

Single file or multiple files?

Treat skills like software

Validation checklist

Final takeaway

What Is Token & LLM Cost Optimization

What Is a Token in LLMs? A Developer's Guide to Cost Optimization and Architecture

1. What Is a Token Under the Hood?

The Multilingual Token Penalty

2. Formatting Mechanics: Linebreaks & The "Emoji Tax"

3. Input vs. Output Tokens: Why Output Costs 3x–5x More

4. Why Model Pricing Varies by 100x (Dense vs. MoE Architecture)

1. Architectural Design: Dense Models vs. Mixture-of-Experts (MoE)

Dense Models

2.1 Linear Scan (`Ctrl + F` or `grep`)

A generic `SKILL.md` example