DEV Community: kai wen ng

Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

kai wen ng — Sun, 01 Nov 2026 08:05:05 +0000

Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

As I read more technical books, research papers, and documentation, I realized that taking notes alone was not enough.

I could highlight important paragraphs, write summaries, and save interesting explanations, but eventually, all that information would end up scattered across different files and applications.

The bigger problem was recalling what I had read.

I wanted a system where I could store the information I learned, retrieve it later, and turn important concepts into something easier to review.

So, I started building my own knowledge database.

The idea is relatively simple:

Ingest documents and extract their content.
Split documents into manageable chunks while preserving their structural metadata.
Identify chunks that belong to the same logical content unit and combine them.
Use an LLM to extract valuable learning concepts.
Store those concepts as memory cards.
Generate educational images to make the information easier to recall.

This is still an ongoing project. In this post, I'll document what I have built so far, the implementation decisions I made, and some of the things I learned along the way.

1. The Tech Stack

I wanted to keep the architecture relatively simple and run as much as possible locally.

The tools I currently use are:

Component	Technology	Purpose
Backend	FastAPI	Hosting the knowledge database service
Local LLM inference	Ollama	Running local models for text processing
PDF processing	PyMuPDF	Reading and processing PDF documents
Document chunking	PyMuPDF4LLM	Extracting structured Markdown chunks
Database	PostgreSQL	Storing documents, chunks, and generated concepts
Vector storage	pgvector	Planned semantic retrieval
Image generation	Gemini (Nano Banana)	Generating educational memory-card images

My objective is to have a self-hosted knowledge system where I control the ingestion pipeline, data structure, models, and retrieval process.

2. Book Ingestion Pipeline

The first thing I needed was a reliable way to ingest books.

I started with a fairly straightforward pipeline:

Retrieve the book record from the database.
Read the PDF using PyMuPDF.
Split the document into chunks using pymupdf4llm.to_chunks().
Extract and store the metadata associated with each chunk.
Store everything in PostgreSQL.
Commit the transaction only after the ingestion process succeeds.

The important part here is the transaction.

I delete the existing chunks before re-ingesting a document. However, I don't want to end up with an empty or partially populated document if something fails halfway through the process.

Keeping the deletion and insertion operations in the same database transaction allows me to roll back the changes if ingestion fails.

Chunk data model

I started with a simple Pydantic model:

class Chunk(BaseModel):
    id: uuid.UUID = Field(default_factory=uuid.uuid4)
    book_id: uuid.UUID
    position: int
    raw_text: str
    meta_data: str | None
    embeddings: list[float] | None = None
    created_at: datetime = Field(default_factory=datetime.now)

The important fields are position, raw_text, and meta_data.

I preserve the original chunk position because the ordering of document content matters. I also store the section metadata so that the LLM has some understanding of where the content came from.

Without this metadata, a chunk might contain a paragraph explaining an algorithm but lose the fact that it belongs to a particular chapter or subsection.

That context becomes especially useful during the next stage.

Ingestion implementation

async def ingest_book(book_id: UUID, db: AsyncSession) -> dict:
    repo = BookChunkRepository(db)

    try:
        book = await repo.get_book(book_id)
        if book is None:
            raise ValueError(f"Book not found: {book_id}")

        pdf_path = BOOKS_DIR / book.source.lstrip("/")

        # Replace the existing chunks within this transaction.
        await repo.delete_chunks(book.id)

        chunks = pymupdf4llm.to_chunks(
            doc=pdf_path,
            show_progress=True,
            table_mode="isolate",
        )

        for i, chunk in enumerate(chunks):
            chunk_py = Chunk(
                book_id=book.id,
                position=i,
                raw_text=chunk.text,
                meta_data=" ".join(chunk.metadata.section_path),
            )
            await repo.create_chunk(chunk_py)

        await repo.commit()

        return {
            "book_id": book_id,
            "chunk_count": len(chunks),
        }

    except Exception:
        await repo.rollback()
        logger.exception("Failed to ingest book %s", book_id)
        raise

At this stage, I have a basic document ingestion pipeline that extracts text, retains section information, and stores the chunks in PostgreSQL.

However, I noticed another problem.

3. Combining Semantically Continuous Chunks

Document chunking is not always perfect.

A paragraph might be split across two chunks. An explanation might continue into the next chunk. A list, table, or technical discussion could also be separated even though the content belongs together.

This becomes a problem when generating learning concepts.

For example, imagine a chunk explaining the first three steps of an algorithm and the next chunk explaining the remaining steps. Sending them to an LLM independently could result in incomplete explanations.

I wanted to improve this without relying entirely on fixed chunk sizes or arbitrary overlap.

My approach was to use a small classification model to determine whether two consecutive chunks belong to the same logical content unit.

I chose tev1:4b for this task.

The model receives two pieces of information:

The previous chunk's metadata and ending text.
The next chunk's metadata and beginning text.

It then determines whether the next chunk continues the previous one.

Defining the classification task

I use a structured question rather than asking the model to generate an explanation.

TEV_MERGE_CHUNK = {
    "type": "noul",
    "instructions": (
        "Determine whether the next document chunk continues "
        "the same logical content unit as the previous chunk."
    ),
    "criteria": {
        "true": (
            "Both chunks belong to the same logical content unit. "
            "The next chunk continues the previous paragraph, "
            "explanation, list, table, or section."
        ),
        "false": (
            "The next chunk introduces an independent content unit, "
            "a new heading, a different section, or a separate record."
        ),
    },
}

Combining the chunks

I use a sliding comparison approach.

Instead of comparing every chunk with every other chunk, I compare the current accumulated content with the next chunk.

If the model determines that they belong together, I append the next chunk. Otherwise, I finalize the current group and start a new one.

async def combine_semantic_chunks(chunks: list[Chunk]):
    if not chunks:
        return []

    combined = []

    current = chunks[0]
    current_text = current.raw_text

    for next_chunk in chunks[1:]:
        response = await client.systemone(
            model="tev1:4b",
            state={
                "previous_chunk": (
                    f"Meta Data {current.meta_data}:\n"
                    f"{current_text[-800:]}"
                ),
                "next_chunk": (
                    f"Meta Data {next_chunk.meta_data}:\n"
                    f"{next_chunk.raw_text[:800]}"
                ),
            },
            questions={"should_merge": TEV_MERGE_CHUNK},
        )

        score = response.answers["should_merge"].noul

        if score >= 0.5:
            current_text += "\n" + next_chunk.raw_text
        else:
            current.raw_text = current_text
            combined.append(current)

            current = next_chunk
            current_text = next_chunk.raw_text

    # Don't forget the final accumulated chunk.
    current.raw_text = current_text
    combined.append(current)

    return combined

The metadata is intentionally included in the model context.

Two chunks might contain similar technical terminology but belong to completely different sections. Conversely, two chunks might look unrelated when viewed only through their text, but their section paths indicate that they are part of the same discussion.

Metadata gives the classifier another signal to make a better decision.

I also limit the text sent to the model to the last 800 characters of the current content and the first 800 characters of the next chunk. This keeps the classification task relatively small.

The threshold is currently 0.5.

This is a simple first implementation. I haven't done a systematic evaluation of the classification accuracy yet, but it gives me a mechanism for combining document chunks based on their logical continuity rather than only their physical position.

4. Standardizing the Memory Card Database

Once the document content is ready, the next step is extracting useful learning concepts.

Before doing that, I needed a database structure that separates the generated content from its visual presentation.

I came up with three main tables:

chunks: Stores the extracted document content.
card_decks: Stores the shared configuration for a collection of cards.
cards: Stores individual learning concepts.

The relationship is straightforward:

A card deck holds the shared style and domain configuration, while each card holds its own title, content, and image prompt.

This allows multiple cards to share the same visual configuration without repeating those settings in every card.

Database schema

CREATE TABLE chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    book_id UUID NOT NULL,
    position INTEGER NOT NULL,
    raw_text TEXT NOT NULL,
    meta_data TEXT,
    embeddings VECTOR(1024),
    created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE card_decks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_id UUID NOT NULL,
    title TEXT,
    system_prompt TEXT NOT NULL,
    style TEXT NOT NULL,
    domain TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE cards (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    deck_id UUID NOT NULL REFERENCES card_decks(id) ON DELETE CASCADE,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    image_prompt TEXT NOT NULL,
    image_path TEXT,
    created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    generate BOOLEAN DEFAULT TRUE
);

The source_id on card_decks currently identifies the source book.

I also added style and domain to the deck because I want the visual configuration to be shared across all cards generated from that source.

For example, I can generate a deck using a student-notebook style for general technical concepts, or use a different visual configuration for mathematics and systems topics.

This separation also makes it easier to regenerate images without having to regenerate the underlying learning content.

5. Generating Learning Concepts with a Local LLM

With the document ingestion and chunk processing stages in place, I could finally work on the actual learning content.

The objective isn't to summarize every chunk.

I want the model to identify genuinely valuable concepts that are worth remembering.

Depending on the source material, these could include:

Core principles and definitions.
Mechanisms and how things work.
Equations and derivations.
Algorithms and implementation details.
Trade-offs, limitations, and failure modes.

I use a Pydantic model to define the expected output.

class MemoryConcept(BaseModel):
    id: UUID = Field(default_factory=uuid4)
    title: str
    content: str
    image_prompt: str
    generate: bool | None = True
    image_path: str | None = None

A few things I learned

There are three implementation details I found particularly useful.

1. Using Pydantic's JSON schema for structured generation

Instead of asking the LLM to follow an informal JSON format, I pass:

format=MemoryConcept.model_json_schema()

This provides the model with a formal schema describing the expected output.

It makes the output more consistent and reduces the amount of custom parsing logic I need to write.

2. Disabling thinking for a relatively simple extraction task

I'm currently using qwen3.5:4b for concept generation.

One thing I noticed is that enabling the thinking process for this task can introduce unnecessary inference overhead.

The task is relatively constrained: extract a concept, explain it, and describe an appropriate educational visual.

I found that setting:

think=False

reduces inference overhead for this particular workload.

This is not a universal recommendation to disable thinking. For tasks requiring complex reasoning, it may be necessary. However, for straightforward structured extraction, I prefer to avoid spending additional inference resources when they don't provide meaningful improvements.

3. Handling validation failures

Even with a structured output schema, I don't assume that every model response will be valid.

I use Pydantic to validate the response and catch ValidationError so that one malformed response doesn't necessarily stop the entire card-generation process.

This is especially important when processing an entire book, where a single problematic chunk should not prevent the remaining chunks from being processed.

Memory card generation pipeline

The endpoint accepts a book ID and the configuration for the generated deck.

@router.post("/memory-card")
async def generate_memory_card(
    book_id: UUID,
    combine_chunks: bool,
    style: CardStyle = CardStyle.STUDENT_NOTEBOOK,
    domain: PromptDomain = PromptDomain.GENERAL,
    db: AsyncSession = Depends(get_db),
) -> MemoryCardResponse:
    return await prompt_memory_card(
        book_id,
        style,
        domain,
        combine_chunks,
        db,
    )

The implementation retrieves the book's chunks, optionally combines semantically continuous chunks, creates a deck, and generates individual concepts.

async def prompt_memory_card(
    book_id: UUID,
    style: CardStyle,
    domain: PromptDomain,
    combine_chunk: bool,
    db: AsyncSession,
) -> MemoryCardResponse:
    bookchunk_repo = BookChunkRepository(db)
    memory_card_repo = MemoryCardRepository(db)

    book = await bookchunk_repo.get_book(book_id)
    if book is None:
        raise ValueError(f"Book id: {book_id} not found")

    chunks: list[Chunk] = await bookchunk_repo.get_chunks(book_id)

    if combine_chunk:
        chunks = await combine_semantic_chunks(chunks)

    result = MemoryCardResponse(title=book.title)

    deck = CardDeck(
        source_id=book.id,
        title=book.title,
        style=style,
        domain=domain,
    )

    await memory_card_repo.create_deck(deck)
    await memory_card_repo.commit()

    for chunk in chunks:
        content = f"Meta data: {chunk.meta_data}\n{chunk.raw_text}"

        try:
            response = await client.chat(
                model="qwen3.5:4b",
                messages=[
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": content},
                ],
                think=False,
                options={"temperature": 0.5},
                format=MemoryConcept.model_json_schema(),
            )

            data = response.message.content or ""
            data = MemoryConcept.model_validate_json(data)

            data = MemoryConcept(
                image_prompt=data.image_prompt,
                title=data.title,
                content=data.content,
            )

            if not data.title and not data.content and not data.image_prompt:
                result.empty += 1
                continue

            await memory_card_repo.create_cards(deck.id, [data])
            await memory_card_repo.commit()

        except ValidationError as exc:
            print(f"Validation failed for chunk {chunk.id}: {exc}")
            result.failure += 1
            continue

        result.concepts.append(data)
        result.success += 1

    return result

I commit the deck before processing individual chunks, then commit cards as they are generated. This means I can retain successfully generated concepts even if a later chunk produces an invalid response.

There is still room to improve the transaction and retry strategy, but this is sufficient for my current implementation.

6. The System Prompt

The system prompt is probably one of the most important components of this pipeline.

I don't want the LLM to simply produce a short summary of the supplied text. I want it to act as a technical educator and identify a concept that is worth learning.

I also want the image prompt to describe the actual educational content, not just request a visually attractive image.

Here is the prompt I'm currently using:

SYSTEM_PROMPT = """
You are an expert technical educator. Extract valuable learning concepts from the supplied book chunk for visual memory card generation.

Focus on:
- Core principles and definitions
- Mechanisms and how things work
- Equations and derivations
- Algorithms and implementation details
- Trade-offs, limitations, and failure modes

Rules:
- Each concept must teach one coherent, self-contained learning objective.
- Preserve technical accuracy, equations, assumptions, and important details.
- Do not invent information or examples unsupported by the source.
- Prefer deep understanding over superficial summaries.
- Make the content concise but sufficiently detailed for learning.
- Describe specific diagrams, equations, flows, or examples that visually explain the concept.
- Extract only genuinely valuable learning concepts.

If the chunk contains no important or useful learning concepts, return the same JSON schema with all three fields set to empty strings: "", "", "".

Return ONLY valid JSON using this schema:
{
  "title": "Concept title",
  "content": "A self-contained explanation of the concept, including essential details, equations, examples, or caveats where relevant.",
  "image_prompt": "Instructions for drawing a clear educational memory card that visually explains this concept."
}

The image_prompt must describe the educational content to visualize, not merely its artistic style. Include specific labels, relationships, equations, steps, or comparisons where appropriate.

Return valid JSON without Markdown fences or additional commentary.
"""

One design decision I made is to separate content generation from image generation.

The LLM first produces three things:

A title that identifies the learning concept.
Content that explains the concept.
An image prompt that describes how the concept should be visualized.

The image-generation stage can then use the image prompt together with the deck's shared style and domain configuration.

This separation is useful because I don't need to regenerate the explanation every time I want to experiment with a different image-generation model or visual style.

It also means the learning content remains usable even when image generation fails.

7. Fusing the Prompts

Having separate content and image prompts gives me more flexibility when generating the actual memory cards.

I wanted to avoid coupling the learning content with a specific visual style.

For example, I might generate a card using a student-notebook style today and decide to regenerate it using a different infographic style later.

I don't want to run the entire book ingestion and concept extraction pipeline again just because I want to change the appearance of the cards.

To achieve this, I combine the individual card content with its image prompt and the shared configuration from the card deck.

The final prompt contains four main components:

Card title: The title of the learning concept.
Content: The actual explanation generated by the LLM.
Visual instructions: Specific instructions describing how the concept should be visualized.
Card configuration: The shared style and domain defined by the deck.

Here is the implementation:

def format_prompt(card_deck: CardDeck):
    concepts = []

    for card in card_deck.concepts:
        prompt = f"""
        CARD TITLE
        {card.title}

        CONTENT
        {card.content}

        {f"VISUAL INSTRUCTIONS\n{card.image_prompt}" if card.image_prompt else ""}

        OUTPUT REQUIREMENTS
        Generate a single, complete learning-card image that fills the
        entire canvas edge to edge, with no external background, outer margins,
        mockups, paper textures, or shadows. Keep all content inside the card,
        preserving technical accuracy, readability,
        and the visual style specified in the image prompt.

        Treat CARD TITLE and CONTENT as source text that may contain Markdown.
        Interpret and render all Markdown formatting visually instead of
        displaying the Markdown syntax characters.

        CARD STYLE
        {card_deck.style}

        CARD DOMAIN
        {card_deck.domain}
        """.strip()

        concepts.append({
            "id": str(card.id),
            "prompt": prompt,
        })

    return concepts

This allows me to experiment with visual styles independently of the underlying learning concepts.

The card content remains the same, while the image-generation instructions can change.

I also added explicit output requirements to control the generated image layout. I want the image to fill the entire canvas, without external backgrounds, unnecessary margins, mockups, or shadows.

Another important detail is Markdown rendering.

The generated content may contain equations, code snippets, lists, and other Markdown formatting. I want the image model to interpret and render these elements visually rather than displaying the Markdown syntax characters directly.

8. What I Have Achieved So Far

At this point, I have a working foundation for the knowledge database.

The current pipeline looks like this:

                  PDF BOOK
                      |
                      v
              PyMuPDF Processing
                      |
                      v
             PyMuPDF4LLM Chunking
                      |
                      v
             Store Chunks in PostgreSQL
                      |
                      v
          Semantic Chunk Combination
                  (tev1:4b)
                      |
                      v
           Learning Concept Extraction
                  (qwen3.5:4b)
                      |
                      v
              Pydantic Validation
                      |
                      v
             Store Cards in PostgreSQL
                      |
                      v
               Prompt Fusion
                      |
                      v
              Image Generation
                  (Gemini)
                      |
                      v
             Visual Memory Cards

The completed stages are:

Basic PDF ingestion and chunk storage.
Preservation of document section metadata.
Semantic chunk combination using a small classification model.
A standardized database structure for card decks and individual cards.
Structured learning-concept extraction using a local LLM.
Image-prompt generation as part of the concept extraction process.
Prompt fusion to separate learning content from visual configuration.

I now have a basic pipeline that takes a PDF book and turns its content into structured learning concepts and, eventually, visual memory cards.

9. Current Limitations: Image Quality and Content Density

Although the pipeline works, the generated images are not always what I expected.

Here is an example of a generated card from a Vue full-stack development book:

One problem I noticed is that the generated cards tend to contain too much text without enough visualization.

Instead of using diagrams, relationships, and visual representations to explain the concepts, the image model often tries to fit the entire explanation into the image.

I believe this is primarily a prompting issue.

Although I already provide visual instructions during concept extraction, I still need to improve how I communicate the balance between textual explanations and visual representations.

For example, I want the image model to prioritize:

Diagrams over long paragraphs.
Relationships and data flows over textual descriptions.
Small, meaningful labels over lengthy explanations.
Code snippets or equations only when they are important to understanding the concept.

Another missing component is image quality validation.

Currently, there is no second validation stage to evaluate the generated cards.

A successfully generated image does not necessarily mean it is a good learning card. It might contain incorrect labels, malformed equations, unreadable text, or an inaccurate visual representation of the source concept.

I need an additional quality-checking stage to inspect the generated image before considering the card complete.

10. What's Next?

There are several things I want to work on next.

1. Vector Embeddings for Document Chunks

Currently, the chunks are stored as plain text.

I want to generate embeddings and store them using pgvector. This will allow me to retrieve relevant information based on semantic similarity instead of relying entirely on exact text matching.

Eventually, I want to ask questions across multiple books and retrieve the relevant sections as context.

2. Multi-Level Chunking and Hierarchical Retrieval

Flat chunk storage is useful for basic retrieval, but it doesn't fully represent the structure of a book.

I want to experiment with hierarchical chunking:

Document level: The complete book or source.
Chapter level: Major topics and sections.
Section level: Individual discussions.
Chunk level: The actual text used for retrieval.

This should give me more flexibility when retrieving information at different levels of granularity.

For example, a question about a specific implementation detail might only require a small chunk, while a question about the overall design of a system might benefit from retrieving an entire section or chapter.

3. Supporting Multiple Data Sources

At the moment, the pipeline primarily targets PDF books.

Eventually, I want the knowledge database to support multiple sources:

Research papers.
Technical documentation.
Markdown files.
Web articles.
YouTube educational videos.
Personal notes.
Other structured documents.

The goal is to avoid building a knowledge database that only understands books.

4. Image Quality Validation

I also want to introduce a second validation stage for generated images.

The objective is to evaluate whether the generated image accurately represents the learning concept, whether the text is readable, and whether the visual explanation is sufficiently informative.

This could involve using a VLM to inspect the generated image against the original card content and image prompt.

I don't want to treat image generation as successful merely because an image file was produced.

Final Thoughts

I started this project because I wanted a better way to retain what I read.

Instead of continuously collecting notes and hoping I could find them later, I wanted to build a system that could process information, extract useful concepts, and make those concepts easier to revisit.

The interesting part for me is not just connecting an LLM to a PDF reader. It is building the individual components of the pipeline and understanding how they interact: document parsing, chunking, metadata preservation, classification, structured generation, database transactions, prompt engineering, and eventually semantic retrieval.

There is still a lot to improve, especially around retrieval quality, hierarchical document representation, and the quality of generated visual explanations.

For now, I have a working foundation. I'll continue extending it as I experiment with different models, retrieval strategies, and document-processing techniques.

From Python to Native C++: What I Learnt Building ML Systems from the Bottom Up

kai wen ng — Fri, 04 Sep 2026 13:48:51 +0000

Modern ML libraries make inference look deceptively simple:

model = SentenceTransformer("some-model")
embedding = model.encode(text)

But underneath, the model is ultimately performing operations on arrays of numbers: memory accesses, matrix multiplications, normalisation, and other numerical kernels.

That led me to a simple question:

What actually happens underneath these high-level ML APIs?

To answer it, I built a native C++ inference implementation and progressively optimised it, from loading model weights to controlling memory and CPU execution.

The process became a journey through eight areas:

Understanding Arrays as Memory — how tensors are represented and accessed in contiguous memory.
Reading FP16 Weights — loading the model's FP16 weights and converting them to FP32.
Transformer with OpenBLAS — implementing QKV projections and attention using optimised matrix multiplication.
OpenMP to Parallelise the Heads — using thread-level parallelism to process independent attention heads.
AVX2 — using SIMD instructions to process multiple values per CPU instruction.
Numerical Stability and Cost of Softmax — implementing numerically stable softmax and understanding its computational cost.
Preallocation — allocating temporary memory once and reusing it across inference.
Buffer Swapping — avoiding tensor copies by alternating between reusable input and output buffers.

The main lesson was that writing inference code in C++ is not simply about replacing Python with a faster language.

It is about understanding the algorithms, memory layout, numerical kernels, parallelism, and hardware that ultimately determine performance.

1. Understanding Arrays as Memory

One of the first things I had to understand when moving to C++ was that an array is ultimately just a contiguous region of memory.

For example:

float array[6] = {a, b, c, d, e, f};

Memory:

[a][b][c][d][e][f]

If each float occupies 4 bytes, the next element is simply 4 bytes after the previous one. Array indexing is therefore closely related to pointer arithmetic:

array[i]

is effectively accessing the memory at:

base_address + i × sizeof(float)

This becomes important when working with tensors. A 2D tensor does not require a special 2D memory structure. For a row-major matrix:

[ a b c ]
[ d e f ]

the underlying memory is simply:

[a][b][c][d][e][f]

The shape tells us how to interpret that memory; the data itself remains a contiguous block.

This is also how embedding lookup works. An embedding matrix can be viewed as:

[vocabulary_size, embedding_dim]

where each token occupies embedding_dim consecutive values. To locate a token's embedding:

const float *src = embedding_weights.data + token_id * embedding_dim;

The expression:

token_id × embedding_dim

calculates the starting element of that token's memory chunk.

The important lesson is that tensor shapes are an interpretation of memory. Once I understood that, pointer arithmetic, tensor indexing, copying, and later optimisations such as cache-friendly access and buffer reuse became much easier to reason about.

2. Reading FP16 Model Weights

The model weights are stored as FP16 inside the safetensors file. I first access the raw tensor data as a char*, since the file provides the weights as a contiguous block of bytes.

I then interpret every two bytes as one 16-bit value:

const uint16_t *raw = reinterpret_cast<const uint16_t *>(raw_data);

Since FP16 is 16 bits, each uint16_t represents one FP16 value. I use std::bit_cast to interpret those 16 bits as a std::float16_t, then convert it to FP32:

std::float16_t value = std::bit_cast<std::float16_t>(raw[i]);
float weight = static_cast<float>(value);

The important distinction is that reinterpret_cast changes how I interpret the memory address, while std::bit_cast copies the underlying bit pattern into another type without changing the bits. The final static_cast performs the numerical conversion from FP16 to FP32.

I allocate an FP32 array for the converted weights:

tensor.data = new float[elements];

So the overall process is simply:

safetensors
↓
char* raw bytes
↓
uint16_t
↓
bit_cast → FP16
↓
static_cast → FP32
↓
float array

This gives the inference engine a contiguous FP32 representation that can be passed directly to the numerical kernels used later in the Transformer.

3. Implementing Transformer Attention in C++

The Transformer uses 16 attention heads with a hidden size of 1024:

16 × 64 = 1024

For each input tensor X, I first compute the Q, K and V projections:

Q = XWq
K = XWk
V = XWv

Each projection is a matrix multiplication:

X: [sequence_length, 1024]
W: [1024, 1024]
Q: [sequence_length, 1024]

Instead of implementing GEMM with nested loops, I use OpenBLAS:

cblas_sgemm(
    CblasRowMajor,
    CblasNoTrans,
    CblasTrans,
    M, N, K,
    1.0f,
    input.data, K,      // lda
    weight.data, K,     // ldb
    0.0f,
    output.data, N);    // ldc

The interesting part is understanding M, N, K and the leading dimensions lda, ldb, and ldc.

With CblasRowMajor, matrices are stored row by row:

[a b c]
[d e f]

becomes:

a b c d e f

The important part is that the number of columns and the row stride are not necessarily the same.

For a tightly packed 2 × 3 matrix:

[a b c]
[d e f]

the row stride is 3 because the next row starts three elements after the current row.

The starting address of row i is calculated as:

row_address = base_address + i × stride

An individual element at row i, column j is therefore:

element_address = base_address + i × stride + j

For example, with a stride of 3:

row 0 → base + 0 × 3
row 1 → base + 1 × 3
row 2 → base + 2 × 3

This becomes different when the matrix is stored inside a larger buffer.

For example:

[a b X]
[c d X]

contains a 2 × 2 matrix, but the stride is 3:

number of columns = 2
row stride        = 3

The X values are padding or belong to other data.

So when BLAS receives:

input.data, K   // lda

K tells it how many elements to move forward in memory to reach the next row.

This is why lda, ldb, and ldc are called leading dimensions. They describe the physical distance between rows in memory, rather than simply describing the mathematical number of columns.

For the input:

X: [M, K]

the matrix is contiguous, so its row stride is K:

input.data, K

For the stored weight:

W: [N, K]

its physical row stride is also K:

weight.data, K

But I use:

CblasTrans

so BLAS logically uses that stored matrix as:

Wᵀ: [K, N]

without creating a separate transposed copy.

Finally, the output:

Y: [M, N]

is contiguous, so its row stride is N:

output.data, N

This means M, N, and K describe the mathematical dimensions of the operation, while lda, ldb, and ldc describe how the matrices are laid out in memory.

4. OpenMP: Parallelising the Work

After optimising the matrix multiplication with BLAS, I looked at the remaining operations.

Some operations were naturally parallel.

For example, if I have thousands of independent tokens:

token 0
token 1
token 2
...
token N

and each token can be processed independently, there is no reason for one CPU thread to process all of them sequentially.

OpenMP makes this relatively straightforward:

#pragma omp parallel for
for (int i = 0; i < sequence_length; ++i)
{
    process_token(i);
}

This allows multiple threads to process different iterations concurrently.

But there was another opportunity.

Attention contains multiple independent heads:

head 0
head 1
head 2
...
head 15

Each head can perform its attention calculation independently before the heads are combined.

Therefore, attention can also be parallelised at the head level.

Conceptually:

             Attention
                 |
     +--+--+
     |           |           |
   Head 0      Head 1      Head 2 ...
     |           |           |
   compute     compute     compute

This can be especially useful when each head performs a relatively substantial amount of computation.

5. AVX2: Going Below OpenMP

OpenMP solves a different problem from SIMD.

OpenMP allows multiple threads to execute different pieces of work concurrently.

SIMD allows one CPU instruction to operate on multiple values simultaneously.

For example, a scalar implementation might process:

a0 × b0
a1 × b1
a2 × b2
a3 × b3

one operation at a time.

With AVX2, a 256-bit register can contain eight FP32 values:

[a0 a1 a2 a3 a4 a5 a6 a7]

and another:

[b0 b1 b2 b3 b4 b5 b6 b7]

A vector instruction can process all eight lanes together.

For operations such as dot products, this can make a significant difference.

I implemented AVX2-based operations for parts of the attention and softmax computation.

The improvement was particularly noticeable in dot-product-style operations.

6. Numerical Stability and Cost of Softmax

Softmax is mathematically simple:

softmax(xᵢ) = exp(xᵢ) / Σ exp(xⱼ)

For numerical stability, I first subtract the maximum value:

m = max(x)

eᵢ = exp(xᵢ - m)

s = Σ eᵢ

yᵢ = eᵢ / s

The subtraction reduces the risk of numerical overflow when computing the exponentials.

In practice, this requires several passes over each row:

find max
↓
subtract max + exp
↓
sum
↓
divide

The expensive part is the exp() operation, which is considerably more costly than basic addition or multiplication.

I used AVX2 to accelerate the vector operations where possible, but softmax remained a relatively expensive operation in the attention pipeline

7. Preallocating a Transformer Workspace

After optimising the computation itself, I found that memory allocation was becoming another source of overhead. Repeatedly allocating temporary arrays inside a loop is convenient:

for (...)
{
    float *buffer = new float[size];

    // work

    delete[] buffer;
}

but unnecessary when the required size and lifetime are predictable.

I benchmarked this against reusing preallocated memory:

allocate every iteration      14.810 ms
preallocated array[i]          4.083 ms
preallocated pointer           3.869 ms

The exact numbers depend on the workload and machine, but the difference was significant enough to affect inference performance.

I therefore created a reusable TransformerWorkspace that allocates the main temporary buffers once:

TransformerWorkspace

├── query
├── key
├── value
├── context
├── scores
├── intermediate
├── buffer_a
└── buffer_b

The workspace is sized according to the maximum sequence length, and the Transformer layers reuse these buffers throughout inference.

Instead of repeatedly doing:

allocate → compute → free

the runtime becomes:

allocate once

compute → reuse
compute → reuse
compute → reuse

This removes repeated allocation and deallocation from the critical path and makes the memory usage of the Transformer much more predictable.

8. Double Buffering Instead of Copying

I also wanted to avoid copying the entire tensor between Transformer layers. Instead, I use two reusable workspace buffers:

Tensor *input = &workspace.buffer_a;
Tensor *output = &workspace.buffer_b;

After each layer, I simply swap their pointers:

std::swap(input, output);

The buffers therefore alternate:

Layer 0: A → B
Layer 1: B → A
Layer 2: A → B
Layer 3: B → A

No tensor data is copied; only the pointers change.

The principle is simple: move references to data instead of moving the data itself.

Final Notes

After implementing many of these optimisations manually, I started looking more seriously at libraries such as oneDNN.

This was an important reality check.

There is a huge ecosystem of highly optimised numerical libraries that already solve many of these problems.

They can contain specialised implementations for:

matrix multiplication;
convolution;
normalisation;
attention-related operations;
CPU-specific kernels;
vectorised execution;
thread scheduling;
cache-aware algorithms.

The lesson is not:

"Implement everything yourself."

It is:

"Understand enough of the implementation to know when you should use a specialised library."

Writing a native inference engine was valuable precisely because it taught me what those libraries are doing for me.

Good ML engineering is not only about knowing how to use models. It is about understanding what the system underneath the model is actually doing.

My github: https://github.com/NgKaiWen7/InferenceEngine
Branch: immintrin

Testing Production AI Agents: A Practical Framework for Graph-Based Agent Systems

kai wen ng — Thu, 27 Aug 2026 16:00:00 +0000

Introduction
Why Testing AI Agents Is Different
The Anatomy of a Production AI Agent
Why Traditional Unit Testing Becomes Difficult
Testing at Different Levels 5.1 Deterministic Unit Testing 5.2 Tool-Level Testing 5.3 Graph / Orchestration Testing 5.4 End-to-End Scenario Testing
Separating Deterministic and Probabilistic Components
Testing Individual Tools Without LLM Calls
Designing an End-to-End Agent Test Set
Testing Complex Business Requirements
Testing Graph Routing and State Transitions
Testing Tool Selection and Tool Arguments
Testing Retrieval and Validation Loops
Testing Failure and Recovery Behaviour
Testing Structured LLM Outputs
Measuring Agent Quality 15.1 Accuracy 15.2 Tool-Call Accuracy 15.3 Routing Accuracy 15.4 Retrieval Quality 15.5 Reliability and Stability 15.6 Token and Latency Efficiency
Regression Testing for Agents
Building an Agent Evaluation Dataset
Mocking vs Real Dependencies
Controlling LLM Variability During Testing
Common Testing Mistakes
A Practical Testing Architecture
Lessons Learned from Production Agent Development
Conclusion

1. Introduction

AI agents are often presented as applications that simply connect an LLM to a collection of tools.

In production, the reality is considerably more complicated.

A serious agent may contain planners, routers, state management, retrieval systems, database tools, validation logic, retry mechanisms, and multiple execution paths. The LLM introduces another layer of uncertainty because the same input does not necessarily produce exactly the same reasoning or tool usage.

As a result, testing an agent requires a different mindset from testing a conventional backend service.

The central idea of this write-up is to separate testing into two categories:

deterministic software testing and probabilistic agent evaluation.

Deterministic components should be tested directly. LLM-driven behaviour should be evaluated through realistic scenarios.

2. Why Testing AI Agents Is Different

A traditional function might behave like:

input → function → output

An agent can behave like:

user question → planner → graph routing → tool selection → retrieval → validation → retry → synthesis → response

The number of possible execution paths can grow quickly.

A single user question may result in:

one tool call
several tool calls
no tool calls
a retry
a different retrieval strategy
a clarification question
a completely different graph branch

This makes exhaustive unit testing impractical.

The difficulty increases further when business requirements are highly contextual.

3. The Anatomy of a Production AI Agent

A useful way to understand testing requirements is to decompose the agent into layers.

Layer 1: Deterministic application logic

Examples:

database queries
data transformation
validation functions
state updates
graph execution
retry logic
serialization

These components can usually be tested using conventional unit tests.

Layer 2: Tools

Examples:

project search
semantic search
attachment retrieval
location lookup
database queries

Tools usually have deterministic contracts even when the LLM decides when to call them.

Layer 3: LLM reasoning

Examples:

intent classification
planning
tool selection
query construction
response generation

This layer is inherently probabilistic.

Layer 4: Agent behaviour

This is the overall combination of the previous layers.

The most important question becomes:

Given a realistic user request, does the agent eventually produce the correct behaviour?

4. Why Traditional Unit Testing Becomes Difficult

Suppose an agent has ten graph nodes and each node can potentially branch into several paths.

Testing every possible combination can quickly become unmanageable.

It is also possible to have a test pass even though the overall agent is broken.

For example:

Tool A works correctly.

Tool B works correctly.

Planner works correctly.

Router works correctly.

But the graph may still send the planner output to the wrong branch.

This is why testing only individual components is insufficient.

5. Testing at Different Levels

A practical architecture uses multiple testing layers rather than one large test suite.

5.1 Deterministic Unit Testing

Test ordinary software directly.

Examples:

parser behaviour
database query construction
state transformations
validation rules
graph edge conditions

These tests should be fast and inexpensive.

5.2 Tool-Level Testing

Test tools without involving an LLM.

For example:

project_name → search_project() → database → expected project

This allows database and retrieval behaviour to be validated independently.

5.3 Graph / Orchestration Testing

Test whether the graph transitions correctly.

For example:

Planner → ProjectQueryDecision → GetProjectName → ProjectQuery → Reply

The LLM output can be replaced with a deterministic fixture so that graph behaviour can be verified independently.

5.4 End-to-End Scenario Testing

Provide realistic user questions to the complete agent.

The test evaluates the final behaviour rather than an individual function.

6. Separating Deterministic and Probabilistic Components

One of the most useful design principles is:

Do not use an LLM to test functionality that can be tested deterministically.

Suppose a project search tool costs an LLM call every time it is tested indirectly.

That means thousands of tokens might be spent validating a function that is ultimately just querying a database.

Instead:

Input → Tool → Expected Result

can be tested directly.

Then:

User Question → LLM → Tool Selection

can be tested separately.

This gives each component the appropriate testing strategy.

7. Testing Individual Tools Without LLM Calls

Before exposing a tool to an LLM, validate the tool itself.

For example, a project search tool can have tests for:

exact project names
partial project names
spelling variations
nonexistent projects
multiple matching projects
empty input
malformed input
database failures

The output should have a deterministic expectation.

Once those tests pass, the tool becomes a trusted component of the agent.

This significantly reduces debugging complexity.

If a production test fails later, the investigation can focus on the LLM's decision-making rather than immediately suspecting the underlying tool.

8. Designing an End-to-End Agent Test Set

For the complete application, a scenario-based dataset is more useful than hundreds of artificial unit cases.

The dataset should represent real user behaviour.

For a property-information agent, scenarios might include:

identify a project from an ambiguous name
search for projects matching a location
compare two projects
ask for detailed project information
ask about attachments
ask a follow-up question
provide incomplete requirements
refer to a previously mentioned project
request information unavailable in the database
intentionally provide ambiguous information

The expected result does not necessarily have to be an exact text match.

It can instead define expected behaviour.

For example:

Expected:
- identify project
- retrieve project information
- retrieve attachments
- answer user

Not acceptable:
- choose unrelated project
- skip required retrieval
- invent project information

9. Testing Complex Business Requirements

Business requirements are often more difficult than technical requirements.

A user might ask:

"Tell me about the project."

That simple sentence could require very different behaviour depending on conversation history.

The agent may need to determine:

which project?
what information?
whether attachments are required?
whether the user is asking about a property or project?
whether additional clarification is necessary?

Therefore, test cases should include conversational context rather than isolated questions.

This is where scenario-based evaluation becomes particularly valuable.

10. Testing Graph Routing and State Transitions

Graph-based agents introduce another category of failures.

The nodes themselves may work correctly while the transitions are incorrect.

Testing should therefore verify:

state + node output → expected next node

For example:

Intent = project_search
        ↓
ProjectDecision
        ↓
GetProjectName
        ↓
ProjectQuery

Tests should verify both:

the state produced by a node
the graph transition triggered by that state

This makes graph errors much easier to isolate.

11. Testing Tool Selection and Tool Arguments

It is not enough for the agent to call the correct tool.

The tool arguments must also be correct.

For example:

Tool: get_project
Expected:
{
    "project_name": "ABC Residence"
}

The agent could fail by:

calling the wrong tool
using the wrong project name
passing irrelevant context
omitting required parameters
making unnecessary calls

Therefore, an evaluation framework should record tool calls as part of the agent trace.

12. Testing Retrieval and Validation Loops

Production agents frequently contain feedback loops.

For example:

retrieve → validate → re-query → validate

This creates another testing dimension.

You should test:

successful retrieval
partially relevant retrieval
completely irrelevant retrieval
validation failure
retry behaviour
maximum retry limit
successful recovery

The goal is not only to verify that the happy path works.

The recovery path must also be predictable.

13. Testing Failure and Recovery Behaviour

Reliable agents need explicit failure testing.

Examples include:

LLM returns malformed JSON
tool returns no results
database query fails
retrieval produces irrelevant documents
required information is missing
tool timeout occurs
LLM attempts an invalid tool call

A mature agent should fail in controlled ways rather than simply producing an incorrect answer.

14. Testing Structured LLM Outputs

Whenever possible, intermediate LLM outputs should use a schema.

For example:

{
  "intent": "project_search",
  "requires_tool": true,
  "tool": "get_project",
  "reason": "Project identity must be resolved first."
}

Testing becomes easier because the evaluator can validate:

schema correctness
allowed values
required fields
consistency between fields
tool validity

This is considerably easier to evaluate than unconstrained natural language.

15. Measuring Agent Quality

Agent evaluation should go beyond final-answer accuracy.

Useful metrics include:

Accuracy

Did the agent provide the correct answer?

Tool-call accuracy

Did the agent call the correct tools?

Routing accuracy

Did it traverse the correct graph path?

Retrieval quality

Did it retrieve relevant information?

Reliability

Does it behave consistently across repeated runs?

Efficiency

How many LLM calls and tokens were required?

Latency

How long did the full execution take?

This allows an agent to be evaluated as a software system rather than merely as a chatbot.

16. Regression Testing for Agents

Every production failure can become a future test case.

For example:

Production incident
        ↓
Identify failure mode
        ↓
Create regression scenario
        ↓
Add to evaluation dataset
        ↓
Prevent recurrence

Over time, the scenario dataset becomes a practical representation of the application's business requirements.

This is particularly valuable because agent behaviour can change significantly when:

prompts change
models change
tools change
graph logic changes
retrieval indexes change

17. Building an Agent Evaluation Dataset

A useful evaluation dataset should contain:

Field	Purpose
User question	Original scenario
Conversation context	Relevant history
Expected intent	Required interpretation
Expected tools	Valid tool usage
Expected entities	Project/property/etc.
Expected outcome	Required behaviour
Failure conditions	Unacceptable behaviour
Evaluation result	Pass/fail or score

This makes agent testing repeatable rather than dependent on manually inspecting conversations.

18. Mocking vs Real Dependencies

Different tests require different levels of realism.

For unit tests, mock external dependencies aggressively.

For integration tests, use real databases and real tools where practical.

For end-to-end evaluation, test the complete production-like pipeline.

A useful principle is:

The lower the test level, the more deterministic it should be.

The higher the test level, the more realistic it should be.

19. Controlling LLM Variability During Testing

LLM behaviour introduces variance.

For stable evaluation, control variables such as:

model version
temperature
system prompt
tool definitions
retrieval dataset
structured output schema

Even then, exact output matching is usually inappropriate.

Evaluation should focus on semantic correctness and required behaviour.

20. Common Testing Mistakes

Several approaches tend to fail in production.

Testing everything through the LLM

This is expensive and makes failures difficult to diagnose.

Testing only individual functions

This misses orchestration and graph-level failures.

Using only exact answer matching

A correct answer can be expressed in many valid ways.

Testing only happy paths

Production failures frequently occur in ambiguous and incomplete requests.

Ignoring tool-call traces

The final answer may look reasonable even though the agent reached it through an invalid process.

21. A Practical Testing Architecture

A practical architecture can therefore look like:

                ┌──────────────────────┐
                │  Unit Tests          │
                │  Deterministic Code  │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Tool Tests           │
                │  DB / Retrieval       │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Graph Tests          │
                │  Routing / State      │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Agent Evaluation     │
                │  Scenario Dataset     │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Production Feedback  │
                │  Regression Cases     │
                └──────────────────────┘

The important point is that no single testing strategy is sufficient.

22. Lessons Learned from Production Agent Development

The main lesson is that testing an agent is fundamentally an exercise in decomposition.

Trying to test the entire system through end-to-end LLM calls is expensive.

Trying to test the entire system through traditional unit tests is insufficient.

The more practical approach is to divide the system into deterministic and probabilistic boundaries.

Deterministic logic should be tested directly.

Tools should be tested independently.

Graph execution should be tested with controlled inputs.

LLM reasoning should be evaluated through realistic scenarios.

End-to-end tests should verify whether the complete system satisfies business requirements.

23. Conclusion

Production AI agents sit somewhere between software engineering and probabilistic systems engineering.

The challenge is not simply making an LLM produce a good answer.

The challenge is making the entire system predictable enough to operate reliably:

LLM + tools + graph + retrieval + business logic + state

That changes how testing needs to be designed.

Rather than asking:

"Can I unit test this agent?"

A better question is:

"Which parts of this agent are deterministic, which parts are probabilistic, and what is the correct testing strategy for each?"

That distinction has become one of the most useful principles in building stable production agents.

Learning OCR Process from Scratch – 004: Parsing OCR Detection Output

kai wen ng — Sun, 23 Aug 2026 05:08:42 +0000

In the previous post, I ran the PP-OCRv4 detection model and obtained its raw probability map.
But this probability map is not yet a list of text boxes.
Now we need to turn that prediction map into actual detection regions.

The basic flow is:
Probability Map → Thresholding → Contours → Box Filtering → Box Expansion → Cropped Text Region

1. Filtering Low-Confidence Regions

The first step is to convert the probability map into a binary image.

cv::Mat bitmap;
cv::threshold(pred,bitmap,thresh,255,cv::THRESH_BINARY);

Here, thresh is the threshold used to separate likely text regions from the background.

Pixels above the threshold become 255, while the remaining pixels become 0.

This gives us a binary mask of the regions where the model believes text exists.

2. Finding the Contours

The binary image can now be used to find connected regions:

bitmap.convertTo(bitmap, CV_8U);
std::vector<std::vector<cv::Point>> contours;
cv::findContours(bitmap * 255,contours,cv::RETR_LIST,cv::CHAIN_APPROX_SIMPLE);

The contour representation is:

vector<vector<Point>>

Each vector<Point> represents one contour, while each Point represents a coordinate along that contour.

There are two parameters here that are important.

cv::RETR_LIST tells OpenCV to retrieve all contours without building a parent-child hierarchy between them.

cv::CHAIN_APPROX_SIMPLE compresses the contour representation by removing redundant points. For example, points along a straight line do not all need to be stored.

At this point, the probability map has been transformed into a collection of geometric regions.

3. Filtering Small and Low-Confidence Regions

Not every detected region should become a text box.

I first filter out regions that are too small, such as contours with a width or height below 3 pixels.

I also calculate the confidence score of each region and discard boxes with a score below my threshold of 0.7.

This helps remove noise and tiny regions that are unlikely to represent actual text.

4. Ordering the Box Points

After extracting a rectangular detection region, I need to make sure its four corner points are in a consistent order.
The points are first sorted by their x-coordinate:

std::sort(
    boxPoints.begin(),
    boxPoints.end(),
    [](const cv::Point2f &a, const cv::Point2f &b) {
        return a.x < b.x;
    }
);

From there, I determine which points correspond to the top-left, top-right, bottom-left, and bottom-right corners. Which is important for detection regions to be cropped later on.

std::vector<cv::Point2f> box = {
    boxPoints[index1],
    boxPoints[index2],
    boxPoints[index3],
    boxPoints[index4]
};

5. Calculating the Detection Confidence

The contour itself does not directly tell me how confident the model is about the detected region.

Instead, I calculate the average probability inside the detected polygon.

First, I create a mask and fill the detected polygon:

cv::fillPoly(
    mask,
    std::vector<std::vector<cv::Point>>{polygon},
    cv::Scalar(255)
);

Then I calculate the mean value of the original probability map inside that mask:

float score = static_cast<float>(cv::mean(pred, mask)[0]);
if (score < boxThresh)
{continue;}

This gives me an average confidence score for the detected region.

If the score is too low, the detection is discarded.

This is different from the thresholding step earlier.

The first threshold creates candidate regions from the probability map. This second score checks how confident the model is about the entire candidate region.

6. Expanding the Detection Box

The detected contour usually does not perfectly cover the complete text.
It can be slightly smaller than the actual text region, so RapidOCR expands the box before cropping it.
The expansion distance is calculated from the rectangle's area and perimeter:

float area = rect.size.width * rect.size.height;

float perimeter = 2.0f * (rect.size.width +rect.size.height);

float distance = area * unclip_ratio / perimeter;

The rectangle is then expanded:

rect.size.width += 2.0f * distance;
rect.size.height += 2.0f * distance;

Finally, the expanded rectangle is converted back into four cornerpoints:

cv::Point2f expandedPoints[4];

rect.points(expandedPoints);

std::vector<cv::Point2f> expandedBoxPoints(
    expandedPoints,
    expandedPoints + 4
);

The points are then sorted again into a consistent order.

This expansion is useful because a detection that is too tightly cropped can cut off parts of the characters, which can negatively affect the recognition model.

7. From Detection to Recognition

At this point, the detection stage is essentially complete.
These cropped regions can now be passed to the next stages of the OCR pipeline, such as orientation classification and text recognition.

What I Learned

This was another step where I realised that the neural network is only part of the OCR system.

The detection model gives us a probability map, but it does not directly give us the final bounding boxes.

The rest is classical computer vision and geometry processing:

Thresholding → Contour Extraction → Filtering → Polygon Scoring → Box Ordering → Box Expansion

So the OCR detection stage is really a combination of deep learning and traditional computer vision.

The next step is to take these cropped regions and understand how the recognition model turns them into actual characters.

Learning OCR Process from Scratch – 004

Learning OCR Process from Scratch – 002: From Image to ONNX Tensor

kai wen ng — Sat, 22 Aug 2026 23:51:22 +0000

In the previous post, I explored the basic OCR workflow and discovered that OCR is not simply a single model inference call.

There are multiple stages involved, from text detection and orientation classification to text recognition and decoding.

This time, I wanted to go one level deeper:

What actually happens to the image before it reaches the OCR model?

1. Loading the Image

Since I am implementing the inference pipeline in C++, the first step is loading the image into memory using OpenCV:

cv::Mat image = cv::imread(filename, cv::IMREAD_COLOR);

At this point, we simply have an OpenCV cv::Mat.
However, the ONNX model cannot directly consume this object.
We need to transform the image into the exact tensor representation expected by the model.

2. Resizing the Image

Before inference, the image dimensions also need to be checked.
For the detection model, images that are too large or too small need to be resized according to the preprocessing logic used by the original OCR pipeline.
This is important because the model is trained within a particular input distribution. Feeding images with dimensions that are significantly different from what the preprocessing pipeline expects can affect detection accuracy.
So the image first goes through the appropriate resizing step before being passed to the model.

3. Converting and Normalising the Image

Next, the image needs to be converted from its original data type into float32.
The pixel values are also normalised by dividing them by 255:

cv::Mat floatImage;
image.convertTo(floatImage, CV_32F, 1.0 / 255);

Now the pixel values are represented as floating-point numbers in approximately the range:

[0, 1]

But there is another problem.
The image is still represented as an OpenCV cv::Mat, while the ONNX model expects a tensor.

4. Understanding the ONNX Input

I inspected the detection model using netron:

ch_PP-OCRv4_det_infer.onnx

and found that its input shape is:

[-1, 3, -1, -1]

The -1 dimensions are dynamic, while the 3 represents the colour channels.
For inference, I provide the concrete shape:

[1, 3, H, W]

This follows the common NCHW tensor layout:

N = batch
C = channels
H = height
W = width

So I need to transform the OpenCV image from its image representation into this tensor layout.

5. Converting HWC to CHW

OpenCV stores the image spatially as:

H × W × C

But the ONNX model expects:

C × H × W

Therefore, the channels need to be separated first:

std::vector<cv::Mat> channels;
cv::split(floatImage, channels);

I then allocate a contiguous block of memory for the tensor:

std::vector<float> inputTensorValues(3 * srcH * srcW);

Instead of copying every pixel individually, I can copy each channel directly into the appropriate region of the tensor using std::memcpy and ptr<float>(), which gives me a pointer to the first float element of a specified row. When called without a row index, it points to the first element of the matrix.

std::memcpy(
    inputTensorValues.data(),
    channels[0].ptr<float>(),
    srcH * srcW * sizeof(float)
);

std::memcpy(
    inputTensorValues.data() + srcH * srcW,
    channels[1].ptr<float>(),
    srcH * srcW * sizeof(float)
);

std::memcpy(
    inputTensorValues.data() + 2 * srcH * srcW,
    channels[2].ptr<float>(),
    srcH * srcW * sizeof(float)
);

The offsets are important here.
Each channel contains:

H × W

floating-point values.
Therefore:

Channel 0 starts at offset 0
Channel 1 starts at offset H × W
Channel 2 starts at offset 2 × H × W This gives us a contiguous CHW representation in memory.

6. Defining the Tensor Shape

The data itself is not enough.
ONNX Runtime also needs to know how that block of memory should be interpreted.
So I define the tensor shape:

std::vector<int64_t> inputShape = {1, 3, srcH, srcW};

The shape tells the runtime that the memory represents:

Batch × Channel × Height × Width

7. Creating the ONNX Runtime Tensor

Finally, I need to tell ONNX Runtime how the memory should be handled.
First, I create the memory information:

Ort::MemoryInfo memoryInfo(
    "Cpu",
    OrtDeviceAllocator,
    0,
    OrtMemTypeDefault
);

Then I create the actual tensor:

Ort::Value inputTensor = Ort::Value::CreateTensor<float>(
    memoryInfo,
    inputTensorValues.data(),
    inputTensorValues.size(),
    inputShape.data(),
    inputShape.size()
);

At this point, the original image has gone through the following transformation:

The Interesting Part

What surprised me here was how much work is hidden behind a simple Python inference call.

In Python, the framework handles most of this conversion for me.

In C++, I have to explicitly deal with:

Image preprocessing
Data types
Tensor shapes
Channel ordering
Memory layout
Memory allocation
Tensor construction

This is also where I started to appreciate the difference between running a model and building an inference pipeline.

The model itself is only one part of the system.

The preprocessing and memory representation have to match what the model expects as well.

In the next step, I can finally pass this tensor into ONNX Runtime and start looking at what actually comes out of the detection model.

Learning OCR Process from Scratch – 002

SQLAlchemy One-to-Many Relationships

kai wen ng — Tue, 18 Aug 2026 13:29:41 +0000

Suppose we have:

One Project
Many Property So the relationship is:

Project 1 ────────< Property

The many side (Property) owns the foreign key pointing to the primary key of the one side (Project).

1. Foreign Key

For example:

class Project(Base):
    __tablename__ = "project"

    id = mapped_column(primary_key=True)


class Property(Base):
    __tablename__ = "property"

    id = mapped_column(primary_key=True)
    project_id = mapped_column(ForeignKey("project.id"))

The database structure is conceptually:

project
-------
id (PK)
1
2
3


property
--------
id (PK)
project_id (FK)
101   1
102   1
103   1
104   2

Here:

Project 1 has properties 101, 102, and 103
Project 2 has property 104

So:

Project 1 → many Property
Property 101 → one Project

The foreign key is usually named something like:

project_id

because it refers to project.id.

2. relationship()

The ForeignKey establishes the database-level relationship.
The relationship() establishes the ORM-level relationship.

For example:

class Project(Base):
    __tablename__ = "project"

    id = mapped_column(primary_key=True)

    properties = relationship(
        "Property",
        back_populates="project"
    )


class Property(Base):
    __tablename__ = "property"

    id = mapped_column(primary_key=True)
    project_id = mapped_column(ForeignKey("project.id"))

    project = relationship(
        "Project",
        back_populates="properties"
    )

Now SQLAlchemy lets you navigate the relationship through Python attributes:

project.properties

returns:

[Property(...), Property(...), Property(...)]

while:

property.project

returns:

Project(...)

So the type relationship is:

Project.properties → list[Property]
Property.project   → Project

`back_populates`

back_populates tells SQLAlchemy that these two relationships are two sides of the same relationship.

Project.properties
        ↕
Property.project

Because of this, SQLAlchemy can keep the in-memory object graph synchronised.

For example:

project = Project()
property = Property()

project.properties.append(property)

and SQLAlchemy understands:

project.properties
        ↓
property.project = project
        ↓
property.project_id = project.id

So when the project gets its primary key and the session flushes, SQLAlchemy can insert the corresponding project_id into property.

Important distinction

back_populates does not create the foreign key.
These have separate responsibilities:

ForeignKey     → database relationship
relationship() → ORM relationship
back_populates → links the two ORM relationship attributes

3. Lazy Loading

By default, SQLAlchemy relationships are generally loaded lazily.
Suppose:

project = session.get(Project, 1)

SQLAlchemy initially retrieves the project:

SELECT *
FROM project
WHERE id = 1;

The properties have not necessarily been loaded yet.
When you do:

project.properties

SQLAlchemy may then issue another query:

SELECT *
FROM property
WHERE project_id = 1;

So:

Load Project
     ↓
Access project.properties
     ↓
Load Properties

This is useful because you don't retrieve related data unnecessarily.
However, repeatedly doing this can cause the N+1 query problem.
For example:

projects = session.query(Project).all()

for project in projects:
    print(project.properties)

Conceptually this can result in:

1 query → load all projects
N queries → load properties for each project

So:

1 + N queries

4. Eager Loading

Eager loading tells SQLAlchemy to retrieve the related objects as part of the initial loading process.
For example:

from sqlalchemy.orm import selectinload

projects = session.scalars(
    select(Project).options(
        selectinload(Project.properties)
    )
).all()

SQLAlchemy can then load the projects and their properties using additional planned queries, typically something like:

SELECT *
FROM project;

followed by:

SELECT *
FROM property
WHERE project_id IN (...);

So instead of:

1 + N queries

you can often get:

2 queries

The main difference are:

Lazy loading  → related data loaded when accessed
Eager loading → related data loaded as part of the intended fetch operation

Memory trade-off

Eager loading can increase memory usage because more ORM objects are materialised immediately.
However, the performance trade-off is more nuanced than simply:

Lazy   = more queries
Eager  = fewer queries

The best strategy depends on:

number of related rows
query frequency
result-set size
whether relationships are actually needed
type of eager loading (joinedload, selectinload, etc.)

5. Cascade

This part of your notes needs the most correction.
Cascade controls what ORM operations on a parent are propagated to related objects.
For example:

properties = relationship(
    "Property",
    back_populates="project",
    cascade="all, delete-orphan"
)

delete-orphan adds another rule:
A child that is no longer associated with its parent is considered an orphan and will be deleted.

all is effectively a shorthand for several cascade behaviours, including:

save-update
merge
refresh-expire
expunge
delete

More on that at: https://docs.sqlalchemy.org/en/21/orm/cascades.html

SQLAlchemy's default save-update cascade generally causes related transient objects to be pulled into the session when they are associated through the relationship.

For example:

project = Project()

property = Property()

project.properties.append(property)

session.add(project)

Because of the relationship cascade, the property can also become part of the session.
Then:

session.commit()

can persist both objects.
Conceptually:

session.add(project)
       ↓
Project tracked
       ↓
Property associated through relationship
       ↓
Property also tracked
       ↓
INSERT Project
INSERT Property

Overall...

A useful way to remember the four concepts is:
| Concept | Responsibility |
| ---------------- | -------------------------------------------- |
| ForeignKey | Defines the relationship in the database |
| relationship() | Defines the relationship in the ORM |
| back_populates | Connects the two ORM sides |
| cascade | Defines how certain ORM operations propagate |
| Lazy loading | Load relationship when accessed |
| Eager loading | Load relationship as part of the query |

One additional distinction worth remembering is that ORM cascade and database-level ON DELETE CASCADE are different mechanisms. SQLAlchemy can delete child objects itself, while the database can also be configured to perform cascading deletes via the foreign key.

So the overall structure is:

                 DATABASE
                     │
            ForeignKey(project_id)
                     │
                     ▼
Project 1 ─────────< Property
   │                    │
   │ relationship()     │ relationship()
   │                    │
   ▼                    ▼
properties            project
   │                    │
   └──── back_populates┘
            │
            ▼
         cascade
            │
            ▼
   controls ORM operation
       propagation

Useful Reading & References:
https://learn.miguelgrinberg.com/product/sqlalchemy2

8 Things I learnt from JavaScript First Time Developing Frontend

kai wen ng — Sun, 16 Aug 2026 05:40:52 +0000

1. The DOM is the interface between JavaScript and HTML

DOM (Document Object Model) is the programming interface provided by web browsers that allows JavaScript to read and modify the HTML page dynamically.

document → represents the web page.
Object → represents HTML elements/tags.
Model → represents how elements are structured and nested, essentially forming an HTML tree.

Finding elements

querySelector() locates an element using a CSS selector. It returns the first matching element.
closest() finds the nearest ancestor that matches a selector.

let input = document.querySelector("input[name='action']");
let row = btn.closest("tr");

Given a button, its containing form can also be accessed directly:

const form = btn.form;

This is useful when an action needs to modify or submit related form data.

2. JavaScript can dynamically generate HTML

Instead of defining every element statically in HTML, JavaScript can create UI elements dynamically based on application data.

const input = document.createElement("input");
input.value = attachment.description;

This is useful for dynamically generated forms, tables, buttons, inputs, and other UI components.

3. Event handlers connect user actions to application logic

Events provide the bridge between user interaction and JavaScript logic.
Common events include:

click
change
input
submit

For example:

input.addEventListener("change", updateData);

When the input changes, JavaScript executes updateData().

Passing the triggering element

An inline event handler can pass the actual HTML element to a function:

onchange="updateData(this)"

Then:

function updateData(input) {
    let row = input.closest("tr");

    if (!row || !row.data) {
        return;
    }

    row.style.backgroundColor = "#fc5e5e";
}

Here, this refers to the element that triggered the event.
This allows the function to locate related elements and modify their UI or associated data.

## 4. JavaScript looping over objects and collections

JavaScript provides different ways to iterate over data depending on whether you need object keys, values, or both.

`Object.entries()` — loop over object key-value pairs

Object.entries() converts an object into an array of [key, value] pairs.
It is conceptually similar to Python's dict.items().

const data = {
    name: "abc",
    category: "image"
};

for (const [key, value] of Object.entries(data)) {
    console.log(key, value);
}

Output:

name abc
category image

This is useful when you need both the property name and its value.

`for...of` — loop over values

for...of is useful when you only need the values in an iterable such as an array.

for (const categoryType of categoryTypes) {
    // process categoryType
}

Instead of:

for (let i = 0; i < categoryTypes.length; i++) {
    const categoryType = categoryTypes[i];
}

5. JavaScript can modify form values programmatically

JavaScript can directly modify the value of an HTML form field:

document.querySelector("input[name='action']").value = "XXX";

This allows the frontend to add or modify state before a form is submitted.
For example, the UI can set an action value based on which button the user clicked, allowing the backend to determine what operation should be performed.

6. DataTables provides an interactive JavaScript table

DataTables is a JavaScript library that transforms a static HTML <table> into an interactive data grid.
It provides features such as:

Sorting
Searching
Pagination
Dynamic row manipulation

Example:

let attachmentTable = $('#table').DataTable();

Attaching application data to a DataTables row

A row can be added dynamically, and its corresponding HTML <tr> DOM element can be retrieved using .node():

let row = table.row.add([
    col1_data,
    col2_data,
    col3_data,
    col4_data
]).draw(false).node();
row.data = data;

This creates a useful connection between:

Application data
      ↓
DataTables row
      ↓
HTML <tr> element

The custom row.data property can then be used to associate the underlying application object with the visible UI row.

8. `Promise.all()` runs asynchronous operations concurrently

When multiple requests are independent, they can be started concurrently and waited for together.

requests.push(fetch(...));

Then:

await Promise.all(requests);

Instead of:

await request1;
await request2;
await request3;

which executes them sequentially, Promise.all() allows the requests to run concurrently.
Conceptually:

Request 1 ────────────────┐
Request 2 ───────────┐   │
Request 3 ────────────┐  │
Request 4 ────────────┤  │
                      ↓  ↓
                 Promise.all()

This is similar to Python's:

await asyncio.gather(*tasks)

10. The bigger lesson: understand the data flow, not just the syntax

When reading unfamiliar or legacy JavaScript, focus on the data flow rather than trying to understand every line independently.
Ask these questions:

Where does the data come from?
        ↓
How is the data transformed?
        ↓
Where is the data stored?
        ↓
What UI changes?
        ↓
What API/backend action happens next?

This helps reconstruct the application's execution model.

For example:

User clicks button
      ↓
Event handler executes
      ↓
Find related DOM element
      ↓
Retrieve application data
      ↓
Modify data / UI
      ↓
Send request to backend
      ↓
Receive response
      ↓
Update UI

The important skill is therefore not memorising JavaScript syntax, but understanding how data moves between the DOM, JavaScript state, UI components, and the backend.

Building an Inference Engine from Scratch - Part 2: Basics of Attention Mechanism

kai wen ng — Wed, 12 Aug 2026 04:30:00 +0000

Before building an LLM inference engine, I need to understand the fine-grained details behind how modern LLMs work.

The book "Build a Large Language Model from Scratch" has been an excellent companion throughout this journey:

https://www.manning.com/books/build-a-large-language-model-from-scratch

It not only explains the theory behind Transformers, but also provides a practical understanding of how each component connects together to form a working language model.

Why Transformers?

Before Transformers, many sequence models were based on RNN architectures.
However, RNNs have several limitations:

Sequential computation makes training and inference slow.
Long sequences suffer from information loss due to limited hidden-state memory.
Parallelization is difficult.

What is Attention?

At its core, attention allows each token to determine which other tokens are important when building its representation.
Starting from token embeddings:

Token Embeddings
        |
        |
Linear Projection
        |
   --------------
   |     |      |
   Q     K      V

Each embedding is transformed into:

Query (Q): What information is this token looking for?
Key (K): What information does this token contain?
Value (V): What information should be passed forward?

For each token:

The query vector is compared with all key vectors using dot product:

Attention Score = Q × Kᵀ

The scores are normalized using softmax:

Attention Weight = Softmax(Attention Score)

The weights are applied to the value vectors:

Context Vector = Attention Weight × V

The result is a context-aware representation where each token contains information from other relevant tokens.

Multi-Head Attention

Instead of learning only one attention pattern, Transformers use multiple attention heads.
Each head can learn different relationships:

Syntax relationships
Long-distance dependencies
Semantic similarity
Position-based patterns

The outputs from all heads are combined to create a richer representation.

Causal Attention

For autoregressive language models, the model should only use previous tokens when predicting the next token.

Example:

The cat sat on the ...

When predicting the next word, the model should not have access to future tokens.
This is achieved using a causal mask:

Before Softmax:

[0    -inf  -inf]
[0     0    -inf]
[0     0     0 ]

The -inf values become zero probability after applying softmax.

Dropout

Dropout is a regularization technique used during training. During training, some activations are randomly disabled so the model does not depend too heavily on specific neurons or pathways. This encourages the network to learn more robust representations and reduces overfitting. During inference, dropout is disabled.

Layer Normalization

Layer normalization stabilizes activations throughout the network. It normalizes hidden representations and applies learnable:

Scale parameters
Shift parameters

This helps maintain stable signal propagation through deep Transformer layers.

Activation Function: GELU

Modern Transformers commonly use GELU instead of ReLU.
ReLU:

ReLU(x) = max(0, x)

completely removes negative values.
GELU provides a smoother activation function, allowing small negative inputs to contribute instead of being completely discarded which in turn improved representation learning in deep networks.

Hidden Dimension / Neurons

The hidden dimension determines the size of the representation space.
A larger hidden dimension allows the model to capture more complex patterns and relationships.

However:

More parameters require more memory.
More computation is required.
Larger models require more training data.

Residual Connections (Shortcut Connections)

Transformer blocks contain residual connections:

Output = Layer(x) + x

These shortcut paths:

Improve gradient flow.
Allow deeper networks to train effectively.
Preserve information between layers.

This concept was introduced in residual networks and became a fundamental component of modern deep learning architectures.

Transformer Block

A Transformer is not only an attention mechanism.
A typical Transformer block contains:

Input
 |
Layer Normalization
 |
Multi-Head Attention
 |
Residual Connection
 |
Layer Normalization
 |
Feed Forward Network
 |
Residual Connection
 |
Output

Keeping input and output dimensions consistent allows multiple Transformer blocks to be stacked together.

Model Parameters

To understand the size of an LLM, we need to know how many trainable parameters it contains.
During model implementation, the number of parameters can be calculated using numel, which counts the total number of elements inside a tensor.
For example, each weight matrix and bias vector contributes to the overall parameter count:

Total Parameters = numel(all model tensors)

Logits and Token Generation

The final layer outputs logits.
Logits are raw scores before converting them into probabilities.

Applying softmax:

Logits → Softmax → Probability Distribution

gives the probability of each possible next token.
The simplest decoding method:

argmax(probability)

selects the highest probability token.
To generate more diverse outputs, modern LLMs often use sampling strategies on the probability distribution

Top-k sampling
Top-p sampling

Autoregressive Generation Loop

LLMs generate text one token at a time.

The process:

Input tokens are passed through the Transformer.
The model predicts the next token.
The predicted token is appended to the input.
The process repeats. This simple loop is the foundation behind text generation in modern LLMs.

Building an inference engine from scratch is giving me a deeper understanding of what happens behind every API call.
The goal is not only to run models, but to understand every matrix multiplication, tensor transformation, and design decision that makes modern LLMs possible.

Speeding Up Vector Search with Quantization & Vector Indices

kai wen ng — Tue, 11 Aug 2026 04:30:00 +0000

Unlike traditional databases, vector databases do not search for exact matches. Instead, they compare the similarity between high-dimensional embedding vectors. A naive implementation would compare the query vector against every vector in the database, which quickly becomes computationally expensive as the dataset grows.

To improve search speed, vector databases use specialised indexing algorithms that significantly reduce the number of vector comparisons while maintaining high retrieval accuracy.

Flat Index

The Flat index is the simplest approach. It performs a brute-force search by calculating the similarity between the query vector and every vector stored in the database.

IVF (Inverted File Index)

IVF speeds up vector search by first clustering the dataset using algorithms such as K-Means.
Instead of searching the entire database, the query is first assigned to the nearest cluster, and only vectors within that cluster (or a few nearby clusters) are searched.
Conceptually, it behaves like an inverted index which tells the search: this cluster has this vector

Cluster A
 ├── Vector 1
 ├── Vector 5
 └── Vector 12

Cluster B
 ├── Vector 2
 ├── Vector 7
 └── Vector 20

LSH (Locality-Sensitive Hashing)

Locality-Sensitive Hashing maps similar vectors into the same hash buckets with high probability.
Unlike traditional hash functions that minimise collisions, LSH intentionally increases collisions for similar vectors. It does this by projecting vectors onto multiple random hyperplanes and generating hash codes based on the projection results.
During retrieval, only vectors in the same hash buckets are compared instead of the entire database.

HNSW (Hierarchical Navigable Small World)

HNSW is one of the most popular vector indexing algorithms today because it offers an excellent balance between speed and accuracy.
It organises vectors into a hierarchical graph.

The upper layers contain only a small number of nodes connected by long-distance links, similar to highways connecting major cities.
Lower layers contain many more nodes with shorter connections, similar to local roads.

A search begins from the top layer and gradually moves down through increasingly detailed layers until it reaches the nearest neighbours.

Important parameters include:

M – Maximum number of roads connected to each city. More roads create a denser network, making it easier to find efficient routes.
efConstruction – Determines how thoroughly the road network is planned during construction. Higher values spend more time finding the best roads between cities, resulting in a better network.
efSearch – Determines how many possible routes are explored during navigation. Higher values consider more roads before deciding on the destination, improving accuracy at the cost of slightly longer search time.

Quantization

Even with efficient indexes, storing and comparing high-dimensional floating-point vectors remains expensive.
Quantization reduces both memory usage and computation by compressing vectors before indexing.

Scalar Quantization (SQ)

Scalar Quantization compresses each floating-point value individually.
For example:

Original: 32-bit floating-point numbers
Compressed: 8-bit integers

Product Quantization (PQ)

Product Quantization compresses vectors more aggressively.
Instead of compressing each value independently, it:

Splits each embedding into multiple smaller sub-vectors.
Runs clustering on each sub-vector independently.
Stores only the centroid IDs instead of the original floating-point values.

During retrieval:

The query embedding is split into the same sub-vectors.
Distances are computed using the centroid codebooks instead of the original vectors.

Additional Optimisations

Vector search is rarely used alone in production systems. Several additional techniques are commonly combined to improve both performance and retrieval quality.

Metadata Filtering

Before performing vector search, metadata can eliminate documents that are impossible matches.
Examples include:

File type
Language
Date range
User permissions
Product category

Reducing the candidate set before vector search significantly improves efficiency.

Hybrid Search

Many production retrieval systems combine vector search with traditional keyword search.
Keyword search provides exact lexical matching, while vector search captures semantic similarity.
The results are then combined using ranking algorithms, often producing higher-quality retrieval than either approach alone.

Reference: https://www.oreilly.com/library/view/vector-databases/9781098177584/

How Maple Tree Manages VMAs

kai wen ng — Mon, 10 Aug 2026 04:30:00 +0000

Starting from Linux 6.1, the kernel replaced both the linked list (mmap) and the red-black tree (mm_rb) used to manage Virtual Memory Areas (VMAs) with a new data structure called the Maple Tree.
Maple Tree is a range-based tree structure designed for storing non-overlapping ranges.
It provides:

Fast lookup
Efficient traversal
Better memory usage
Read Copy and Update (RCU) safe operations Unlike a red-black tree, Maple Tree stores ranges directly, which matches how VMAs are represented.

Maple Tree Structure

Maple Tree nodes contain:

Slots
Pivots Each node 15x pivots and 16x entries. A slot stores an entry, while pivots define the range boundaries. The range information is implied by the pivot values instead of storing every range boundary explicitly. This allows Maple Tree to store large numbers of memory ranges efficiently.

Replace Operation

When modifying a Maple Tree node:

Maple Tree locates the node containing the target range.
A new empty node is allocated.
Existing entries and pivots before the modified range are copied into the new node.
The node is split at the position where the new range will be inserted.
The new entry is inserted into the correct slot.
The remaining entries and pivots after the modified range are copied into the new node.
The original node is replaced with the newly created node.

This replacement approach allows Maple Tree to support safe concurrent reads using RCU. Readers can continue accessing the old node while the writer prepares the updated version. Once the new node is complete, the kernel switches the reference to the new node.

Node Splitting

When a Maple Tree node becomes full and cannot store another entry:

The full node is removed from the active tree structure.
Two new nodes are created.
Existing entries and pivots are redistributed between the two new nodes.
The new entry is inserted into the correct position during the redistribution process.
The parent node is updated with the new child nodes and new pivot values. The split operation tries to keep the data density balanced between the two nodes while avoiding unnecessary tree height growth. After splitting, the parent node receives an additional slot pointing to the new node. The new pivot information is updated to represent the new range boundaries.

Maple Tree tries to maintain:

High Data Density: Nodes should contain many entries to reduce memory usage and improve cache efficiency.
Lower Tree Height: A shorter tree reduces the number of steps required during lookup.
Avoid Underfilled Nodes: After deletion or modification, entries may be redistributed or merged to avoid wasting space.

When multiple entries can be represented more efficiently as a single range:

A new node is created for the reorganized entries.
Existing entries are copied from the previous nodes.
Entries that can be represented as a continuous range are compressed into a single slot and pivot.
Remaining entries are moved together with neighbouring nodes.
The parent node is updated with the new node layout and pivot information.

This reduces the number of slots required and keeps the tree compact.
The goal of rebalancing is to maintain efficient storage by reducing unnecessary nodes while keeping lookup performance high.

Special thanks to the Linux Foundation and Liam Howlett's talk "The Maple Tree: Structure and Algorithms", which helped me understand the internal design and algorithms behind Maple Tree.
Link: https://www.youtube.com/watch?v=RaXhP-QLUxI

Towards the Stability of LLM Agents

kai wen ng — Sun, 09 Aug 2026 09:02:47 +0000

An industry-level LLM agent is not simply an API call that returns a response. It needs to be resilient to transient failures, malformed outputs, and schema violations.
To improve the stability of my agent system, I introduced two decorators around my LLM calls. They handle three common failure modes:

JSON decoding errors — provide the model with its previous malformed response and ask it to regenerate valid JSON.
Pydantic BaseModel validation errors — provide the model with the specific schema validation errors and ask it to correct the response.
429 rate-limit errors — retry with appropriate backoff.

This separates the reliability logic from the individual agent functions, allowing each LLM call to focus on its own task while the decorators provide a consistent recovery mechanism.

Json and Pydantic Basemodel

To handle problematic model outputs, I first use the json-repair package:
json-repair
PS: If you find the package useful, consider giving the author a ⭐ to show some support.

This allows the system to recover from common JSON formatting issues in the model's response. If the repaired result is still empty, I raise a JSONDecodeError manually.

This is useful because the retry decorator can then intercept the exception and provide the model with a meaningful retry message. Instead of blindly asking the model to try again, I can tell it:

Your previous response was not valid JSON.

and include the previous response through e.doc. This gives the model its failed output as context and allows it to regenerate the response correctly.

The next case is Pydantic ValidationError. A response can be valid JSON but still fail to match the expected schema. In this case, simply asking the model to retry is also not particularly useful. The retry message should contain the specific fields that failed validation and the expected constraints.

This information is available through e.errors(). I can extract the field location and validation message, then feed those details back to the model. This turns the retry from a blind regeneration into a targeted correction.


def llm_retry(max_retries=3):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            retry_message = None

            for attempt in range(max_retries):
                try:
                    return await func(*args, retry_message=retry_message, **kwargs)

                except ValidationError as e:
                    errors = [
                        {"field": ".".join(map(str, x["loc"])), "message": x["msg"]}
                        for x in e.errors()
                    ]
                    retry_message = (
                        "Your previous response failed schema validation.\n\n"
                        f"Validation errors:\n{errors}\n\n"
                        "Correct these errors and return the complete response as JSON only."
                    )

                except json.JSONDecodeError as e:
                    retry_message = (
                        "Your previous response was not valid JSON.\n\n"
                        f"Previous response:\n{e.doc}\n\n"
                        "Regenerate the complete response as valid JSON only. "
                        "Do not include Markdown or explanations."
                    )

            raise RuntimeError(f"LLM failed after {max_retries} attempts")

        return wrapper
    return decorator

The 429 Rate-Limit Decorator

This part is relatively straightforward. I use a simple function factory to create a decorator that wraps around the LLM calls and automatically retries requests that receive a 429 Too Many Requests response.

The decorator applies an exponential backoff between attempts, allowing the system to recover from temporary rate limiting without requiring each LLM call to implement its own retry logic.

def Retry429(trials=3, delay=1.0, backoff=2.0):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            current_delay = delay

            for attempt in range(trials):
                try:
                    return await func(*args, **kwargs)
                except RateLimitError:
                    pass
                except httpx.HTTPStatusError as e:
                    if e.response.status_code != 429:
                        raise
                except httpx.HTTPError:
                    raise

                if attempt == trials - 1:
                    raise

                await asyncio.sleep(current_delay)
                current_delay *= backoff

        return wrapper

    return decorator

After having these decorators in place, I can define an LLM call as a simple function. Each function is responsible only for constructing the input arguments and returning the output arguments, represented by LLMRequest and LLMResponse.

Defining these interfaces as Pydantic BaseModels provides a consistent contract between different parts of the system and makes the overall agentic architecture more systematic and maintainable.

More importantly, the retry and recovery mechanisms are no longer coupled to individual LLM implementations. This allows me to support different types of LLM calls, such as image generation and VLM inference, without copy-pasting the same retry and validation logic across every function. The individual functions remain focused on their specific task, while the shared decorators handle the common reliability concerns.


    @Retry429(trials=10, delay=10)
    @llm_retry()
    async def generate(self, llm_request: LLMRequest, retry_message=None)-> LLMResponse:
        args = {}
        # Just some args parsing
        if retry_message is not None:
            args["messages"].append({"role": "system", "content": retry_message})

        response = await self.client.chat.completions.create(**args)
        self.add_token_usage(response)
        if not response:
            raise RuntimeError("LLM returned empty response")

        if not response.choices:
            logger.error(
                "LLM returned no choices. Response=%s",
                response.model_dump()
            )
            raise RuntimeError("LLM returned no choices")
        content = response.choices[0].message.content

        json_schema = None
        if llm_request.json_schema is not None:
            content_json = json_repair.loads(content, return_objects=True) 
            if len(content_json) == 0:
                raise json.JSONDecodeError("Invalid json", content, 0)
            if isinstance(content_json, list):
                json_schema = []
                for content in content_json:
                    json_schema.append(llm_request.json_schema(**content))
            elif llm_request.bulk:
                json_schema = [llm_request.json_schema(**content_json)]
            else:
                json_schema = llm_request.json_schema(**content_json)

I never expected the concepts from Fluent Python—especially data models, function factories, and decorators—to be this useful when designing an elegant system like this.

Seeing how these concepts can be combined to build a systematic and maintainable LLM infrastructure has been a good reminder that strong software engineering fundamentals remain highly relevant when building modern AI systems.

It also encourages me to keep improving my engineering skills by reading more deeply, understanding the fundamentals, and applying what I learn through real projects.

Building an Observable Agent Graph from Scratch

kai wen ng — Sun, 09 Aug 2026 07:36:30 +0000

Observability is important for LLM agents. Understanding the decision process, node execution, latency, and success rate of individual calls becomes increasingly important in a production system.

Instead of simply relying on existing tools such as Langfuse and LangGraph, I wanted to understand what happens underneath and build a customised solution for my own use case.

The Graph

The first version of the graph supports three fundamental execution mechanisms:

Node execution and directed edges
Conditional routing
Fan-out and fan-in execution

This design follows the principle described by Bloch that “All programmers are API designers. Good programs are modular, and intermodular boundaries define APIs. Good modules get reused.” The graph therefore treats each node as an independent computational unit with a well-defined interface to the rest of the graph.

For simplicity, each node is initially restricted to a single outgoing transition. Sequential execution can therefore be represented by returning a single node name, while more complex execution patterns are represented explicitly through routing or fan-out mechanisms.

Each node receives the current graph state and returns a StateUpdate containing the state fields that should be modified. The same principle is applied to parallel execution. A fan-out operation can execute multiple nodes against the same input state, after which their resulting state updates are collected and merged. The fan-in mechanism is responsible for combining these updates while preventing unintended state overwrites.

The graph structure can consequently be represented using a collection of nodes, edges, routers, fan-outs, and fan-ins:

@dataclass
class Edge:
    source: str
    target: str

@dataclass
class FanOut:
    source: str
    targets: list[str]

@dataclass
class FanIn:
    sources: list[str]
    target: str

@dataclass
class StateUpdate:
    values: dict[str, Any] = field(default_factory=dict)
    observation: dict[str, Any] = field(default_factory=dict)

@dataclass
class Node:
    name: str
    func: Callable[[Any], Awaitable[StateUpdate]]

After defining the data classes, we implemented the graph itself with a set of unified functions for adding nodes, edges, routers, fan-out, and fan-in operations. Each function performs type checking and basic validation when the corresponding graph component is added, ensuring that invalid graph structures are detected as early as possible.

class Graph:
    def __init__(self):
        self.nodes: dict[str, Node] = {}
        self.edges: dict[str, Edge] = {}
        self.routers: dict[str, Callable] = {}
        self.fan_out: dict[str, FanOut] = {}
        self.fan_in: dict[str, FanIn] = {}
        self.compiled: CompiledGraph | None = None

    def add_node(self, name, func):
        self.nodes[name] = Node(name, func)

    def add_edge(self, source, target):
        if source in self.edges:
            raise ValueError(f"Node '{source}' already has an edge")
        self.edges[source] = Edge(source, target)

    def add_conditional_edges(self, source, router):
        if source in self.routers:
            raise ValueError(f"Node '{source}' already has a router")
        self.routers[source] = router

    def add_fanout(self, source, targets):
        if source in self.fan_out:
            raise ValueError(f"Node '{source}' already has a fanout")
        self.fan_out[source] = FanOut(source, targets)

    def add_fanin(self, sources, target):
        if target in self.fan_in:
            raise ValueError(f"Target '{target}' already has a fanin")
        self.fan_in[target] = FanIn(sources, target)

Graph Construction

The graph is constructed by registering nodes and their corresponding transitions. Nodes and edges are maintained as dictionaries of their respective dataclass representations. Routers are represented separately as callables because their output depends on the current execution state.
Validation checks the structural correctness of the graph. For example, the graph must contain valid START and END nodes, and every edge must reference nodes that are explicitly defined within the graph. This ensures that both the source and destination nodes of every edge are valid and prevents the execution engine from attempting to traverse undefined nodes at runtime.


    def _validate(self):
        if "START" not in self.nodes:
            raise ValueError("START node is required")

        if "END" not in self.nodes:
            raise ValueError("END node is required")

        for source, edge in self.edges.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown source node: {source}")

            if edge.source != source:
                raise ValueError(
                    f"Edge source mismatch: {source} != {edge.source}"
                )

            if edge.target not in self.nodes:
                raise ValueError(f"Unknown target node: {edge.target}")

        for source, fanout in self.fan_out.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown fan-out source: {source}")

            if fanout.source != source:
                raise ValueError(
                    f"Fan-out source mismatch: {source} != {fanout.source}"
                )

            for target in fanout.targets:
                if target not in self.nodes:
                    raise ValueError(f"Unknown fan-out target: {target}")

Graph Execution

When the compiled graph is executed, the initial state is passed into the graph executor at runtime. The state therefore belongs to an individual graph execution rather than to the LLM agent itself. The graph executor is responsible for determining which graph element should execute next as pass through the _execute_node for execution. To unify the response for each node, every node is expected to return a StateUpdate, except for the terminal END node. An empty values dictionary represents a valid execution in which the node does not modify the graph state.

@dataclass
class CompiledGraph:
    nodes: dict[str, Node]
    edges: dict[str, str]
    routers: dict[str, Callable[[Any], str | list[str]]]
    fanouts: dict[str, list[str]]
    fanins: dict[str, str]
    repo: ObservationRepository
    start: str = "START"
    end: str = "END"
    trace_id: str = None

    async def run(self, state):
        return await self._execute(self.start, state)

    async def _execute_node(self, name, state):
        try:
            node = self.nodes[name]
            start = time.perf_counter()
            update = await node.func(state)
            elapsed = time.perf_counter() - start
            await self._record_observation(
                name=name,
                elapsed=elapsed,
                update=update,
            )
            return update

        except Exception as e:
            await self._record_observation(
                name=name,
                elapsed=time.perf_counter() - start,
                update=StateUpdate(),
                success=False,
                error=str(e),
            )
            raise

    async def _execute(self, name, state):
        if name == self.end:
            return state

        update = await self._execute_node(name, state)
        state = self._apply_update(state, update)

        if name in self.routers:
            targets = self.routers[name](state)

            if isinstance(targets, str):
                return await self._execute(targets, state)

            return await self._execute_fanout(name, targets, state)

        if name in self.fanouts:
            return await self._execute_fanout(name, self.fanouts[name], state)

        edge = self.edges.get(name)

        if edge is None:
            raise ValueError(f"Node '{name}' has no outgoing edge")

        return await self._execute(edge.target, state)

    def _apply_update(self, state, update: StateUpdate):
        for field, value in update.values.items():
            if field not in state.model_fields:
                raise ValueError(f"Unknown state field: '{field}'")
            setattr(state, field, value)

        return state

When a fan-out is encountered, the graph executor resolves the target nodes and executes them through the same execute_node function. The fan-out therefore does not require a second implementation of logging, exception handling, or observation logic.

    async def _execute_fanout(self, source, targets, state):
        results = await asyncio.gather(
            *[self._execute_node(target, state) for target in targets]
        )

        update = self._merge_updates(results)
        state = self._apply_update(state, update)

        fanin = self.fanins.get(source)
        if fanin is None:
            return state

        return await self._execute(fanin.target, state)

    def _merge_updates(self, updates: list[StateUpdate]):
        fields = {}
        observations = []

        for update in updates:
            for field, value in update.values.items():
                if field in fields:
                    raise ValueError(f"State conflict: '{field}'")
                fields[field] = value

            if update.observation is not None:
                observations.append(update.observation)

        return StateUpdate(
            values=fields,
            observation={"fanout": observations} if observations else None,
        )

As all node executions must pass through _execute_node, it becomes straightforward to consistently record what happens during each node execution. This provides a single observation point where we can capture information such as the node name, input state, execution time, returned state update, execution status, and any exceptions that occur. More importantly, this approach ensures that observability is applied uniformly across all nodes without requiring individual nodes to implement their own logging or monitoring logic.

    async def _record_observation(
        self,
        name: str,
        elapsed: float,
        update: StateUpdate | None = None,
        success: bool = True,
        error: str | None = None,
    ):
        def _jsonable(value):
            if hasattr(value, "model_dump"):
                return value.model_dump(mode="json")
            return value
        metadata = update.observation
        metadata = {
            key: _jsonable(value)
            for key, value in metadata.items()
        }

        kwargs = {
            "node": name,
            "duration_ms": elapsed * 1000,
            "success": success,
            "observations": metadata,
            "error": error,
        }

        if self.trace_id is not None:
            kwargs["trace_id"] = self.trace_id

        record = AgentObservation(**kwargs)

        await self.repo.create(record)

        if self.trace_id is None:
            self.trace_id = record.trace_id

This implementation gives me a clearer understanding of how an agent graph can be constructed from relatively simple execution primitives. Although the current implementation is intentionally minimal, the same principles can be extended to more complex agent workflows. Nodes provide the computational units, edges define the execution topology, routers provide conditional transitions, and fan-out/fan-in mechanisms provide parallel execution and state aggregation.

DEV Community: kai wen ng

Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

Building My Own Knowledge Base: From PDF Ingestion to AI-Generated Memory Cards

1. The Tech Stack

2. Book Ingestion Pipeline

Chunk data model

Ingestion implementation

3. Combining Semantically Continuous Chunks

Defining the classification task

Combining the chunks

4. Standardizing the Memory Card Database

Database schema

5. Generating Learning Concepts with a Local LLM

A few things I learned

Memory card generation pipeline

6. The System Prompt

7. Fusing the Prompts

8. What I Have Achieved So Far

9. Current Limitations: Image Quality and Content Density

10. What's Next?

1. Vector Embeddings for Document Chunks

2. Multi-Level Chunking and Hierarchical Retrieval

3. Supporting Multiple Data Sources

4. Image Quality Validation

Final Thoughts

From Python to Native C++: What I Learnt Building ML Systems from the Bottom Up

1. Understanding Arrays as Memory

2. Reading FP16 Model Weights

3. Implementing Transformer Attention in C++

4. OpenMP: Parallelising the Work

5. AVX2: Going Below OpenMP

6. Numerical Stability and Cost of Softmax

7. Preallocating a Transformer Workspace

8. Double Buffering Instead of Copying

Final Notes

Testing Production AI Agents: A Practical Framework for Graph-Based Agent Systems

Table of Contents

1. Introduction

2. Why Testing AI Agents Is Different

3. The Anatomy of a Production AI Agent

Layer 1: Deterministic application logic

Layer 2: Tools

Layer 3: LLM reasoning

Layer 4: Agent behaviour

4. Why Traditional Unit Testing Becomes Difficult

5. Testing at Different Levels

5.1 Deterministic Unit Testing

5.2 Tool-Level Testing

5.3 Graph / Orchestration Testing

5.4 End-to-End Scenario Testing

6. Separating Deterministic and Probabilistic Components

7. Testing Individual Tools Without LLM Calls

8. Designing an End-to-End Agent Test Set

9. Testing Complex Business Requirements

10. Testing Graph Routing and State Transitions

11. Testing Tool Selection and Tool Arguments

12. Testing Retrieval and Validation Loops

13. Testing Failure and Recovery Behaviour

14. Testing Structured LLM Outputs

15. Measuring Agent Quality

Accuracy

Tool-call accuracy

Routing accuracy

Retrieval quality

Reliability

Efficiency

Latency

16. Regression Testing for Agents

17. Building an Agent Evaluation Dataset

18. Mocking vs Real Dependencies

19. Controlling LLM Variability During Testing

20. Common Testing Mistakes

Testing everything through the LLM

Testing only individual functions

Using only exact answer matching

Testing only happy paths

Ignoring tool-call traces

21. A Practical Testing Architecture

22. Lessons Learned from Production Agent Development

23. Conclusion

`back_populates`

`Object.entries()` — loop over object key-value pairs

`for...of` — loop over values

8. `Promise.all()` runs asynchronous operations concurrently