DEV Community: kai wen ng

Testing Production AI Agents: A Practical Framework for Graph-Based Agent Systems

kai wen ng — Thu, 27 Aug 2026 16:00:00 +0000

Introduction
Why Testing AI Agents Is Different
The Anatomy of a Production AI Agent
Why Traditional Unit Testing Becomes Difficult
Testing at Different Levels 5.1 Deterministic Unit Testing 5.2 Tool-Level Testing 5.3 Graph / Orchestration Testing 5.4 End-to-End Scenario Testing
Separating Deterministic and Probabilistic Components
Testing Individual Tools Without LLM Calls
Designing an End-to-End Agent Test Set
Testing Complex Business Requirements
Testing Graph Routing and State Transitions
Testing Tool Selection and Tool Arguments
Testing Retrieval and Validation Loops
Testing Failure and Recovery Behaviour
Testing Structured LLM Outputs
Measuring Agent Quality 15.1 Accuracy 15.2 Tool-Call Accuracy 15.3 Routing Accuracy 15.4 Retrieval Quality 15.5 Reliability and Stability 15.6 Token and Latency Efficiency
Regression Testing for Agents
Building an Agent Evaluation Dataset
Mocking vs Real Dependencies
Controlling LLM Variability During Testing
Common Testing Mistakes
A Practical Testing Architecture
Lessons Learned from Production Agent Development
Conclusion

1. Introduction

AI agents are often presented as applications that simply connect an LLM to a collection of tools.

In production, the reality is considerably more complicated.

A serious agent may contain planners, routers, state management, retrieval systems, database tools, validation logic, retry mechanisms, and multiple execution paths. The LLM introduces another layer of uncertainty because the same input does not necessarily produce exactly the same reasoning or tool usage.

As a result, testing an agent requires a different mindset from testing a conventional backend service.

The central idea of this write-up is to separate testing into two categories:

deterministic software testing and probabilistic agent evaluation.

Deterministic components should be tested directly. LLM-driven behaviour should be evaluated through realistic scenarios.

2. Why Testing AI Agents Is Different

A traditional function might behave like:

input → function → output

An agent can behave like:

user question → planner → graph routing → tool selection → retrieval → validation → retry → synthesis → response

The number of possible execution paths can grow quickly.

A single user question may result in:

one tool call
several tool calls
no tool calls
a retry
a different retrieval strategy
a clarification question
a completely different graph branch

This makes exhaustive unit testing impractical.

The difficulty increases further when business requirements are highly contextual.

3. The Anatomy of a Production AI Agent

A useful way to understand testing requirements is to decompose the agent into layers.

Layer 1: Deterministic application logic

Examples:

database queries
data transformation
validation functions
state updates
graph execution
retry logic
serialization

These components can usually be tested using conventional unit tests.

Layer 2: Tools

Examples:

project search
semantic search
attachment retrieval
location lookup
database queries

Tools usually have deterministic contracts even when the LLM decides when to call them.

Layer 3: LLM reasoning

Examples:

intent classification
planning
tool selection
query construction
response generation

This layer is inherently probabilistic.

Layer 4: Agent behaviour

This is the overall combination of the previous layers.

The most important question becomes:

Given a realistic user request, does the agent eventually produce the correct behaviour?

4. Why Traditional Unit Testing Becomes Difficult

Suppose an agent has ten graph nodes and each node can potentially branch into several paths.

Testing every possible combination can quickly become unmanageable.

It is also possible to have a test pass even though the overall agent is broken.

For example:

Tool A works correctly.

Tool B works correctly.

Planner works correctly.

Router works correctly.

But the graph may still send the planner output to the wrong branch.

This is why testing only individual components is insufficient.

5. Testing at Different Levels

A practical architecture uses multiple testing layers rather than one large test suite.

5.1 Deterministic Unit Testing

Test ordinary software directly.

Examples:

parser behaviour
database query construction
state transformations
validation rules
graph edge conditions

These tests should be fast and inexpensive.

5.2 Tool-Level Testing

Test tools without involving an LLM.

For example:

project_name → search_project() → database → expected project

This allows database and retrieval behaviour to be validated independently.

5.3 Graph / Orchestration Testing

Test whether the graph transitions correctly.

For example:

Planner → ProjectQueryDecision → GetProjectName → ProjectQuery → Reply

The LLM output can be replaced with a deterministic fixture so that graph behaviour can be verified independently.

5.4 End-to-End Scenario Testing

Provide realistic user questions to the complete agent.

The test evaluates the final behaviour rather than an individual function.

6. Separating Deterministic and Probabilistic Components

One of the most useful design principles is:

Do not use an LLM to test functionality that can be tested deterministically.

Suppose a project search tool costs an LLM call every time it is tested indirectly.

That means thousands of tokens might be spent validating a function that is ultimately just querying a database.

Instead:

Input → Tool → Expected Result

can be tested directly.

Then:

User Question → LLM → Tool Selection

can be tested separately.

This gives each component the appropriate testing strategy.

7. Testing Individual Tools Without LLM Calls

Before exposing a tool to an LLM, validate the tool itself.

For example, a project search tool can have tests for:

exact project names
partial project names
spelling variations
nonexistent projects
multiple matching projects
empty input
malformed input
database failures

The output should have a deterministic expectation.

Once those tests pass, the tool becomes a trusted component of the agent.

This significantly reduces debugging complexity.

If a production test fails later, the investigation can focus on the LLM's decision-making rather than immediately suspecting the underlying tool.

8. Designing an End-to-End Agent Test Set

For the complete application, a scenario-based dataset is more useful than hundreds of artificial unit cases.

The dataset should represent real user behaviour.

For a property-information agent, scenarios might include:

identify a project from an ambiguous name
search for projects matching a location
compare two projects
ask for detailed project information
ask about attachments
ask a follow-up question
provide incomplete requirements
refer to a previously mentioned project
request information unavailable in the database
intentionally provide ambiguous information

The expected result does not necessarily have to be an exact text match.

It can instead define expected behaviour.

For example:

Expected:
- identify project
- retrieve project information
- retrieve attachments
- answer user

Not acceptable:
- choose unrelated project
- skip required retrieval
- invent project information

9. Testing Complex Business Requirements

Business requirements are often more difficult than technical requirements.

A user might ask:

"Tell me about the project."

That simple sentence could require very different behaviour depending on conversation history.

The agent may need to determine:

which project?
what information?
whether attachments are required?
whether the user is asking about a property or project?
whether additional clarification is necessary?

Therefore, test cases should include conversational context rather than isolated questions.

This is where scenario-based evaluation becomes particularly valuable.

10. Testing Graph Routing and State Transitions

Graph-based agents introduce another category of failures.

The nodes themselves may work correctly while the transitions are incorrect.

Testing should therefore verify:

state + node output → expected next node

For example:

Intent = project_search
        ↓
ProjectDecision
        ↓
GetProjectName
        ↓
ProjectQuery

Tests should verify both:

the state produced by a node
the graph transition triggered by that state

This makes graph errors much easier to isolate.

11. Testing Tool Selection and Tool Arguments

It is not enough for the agent to call the correct tool.

The tool arguments must also be correct.

For example:

Tool: get_project
Expected:
{
    "project_name": "ABC Residence"
}

The agent could fail by:

calling the wrong tool
using the wrong project name
passing irrelevant context
omitting required parameters
making unnecessary calls

Therefore, an evaluation framework should record tool calls as part of the agent trace.

12. Testing Retrieval and Validation Loops

Production agents frequently contain feedback loops.

For example:

retrieve → validate → re-query → validate

This creates another testing dimension.

You should test:

successful retrieval
partially relevant retrieval
completely irrelevant retrieval
validation failure
retry behaviour
maximum retry limit
successful recovery

The goal is not only to verify that the happy path works.

The recovery path must also be predictable.

13. Testing Failure and Recovery Behaviour

Reliable agents need explicit failure testing.

Examples include:

LLM returns malformed JSON
tool returns no results
database query fails
retrieval produces irrelevant documents
required information is missing
tool timeout occurs
LLM attempts an invalid tool call

A mature agent should fail in controlled ways rather than simply producing an incorrect answer.

14. Testing Structured LLM Outputs

Whenever possible, intermediate LLM outputs should use a schema.

For example:

{
  "intent": "project_search",
  "requires_tool": true,
  "tool": "get_project",
  "reason": "Project identity must be resolved first."
}

Testing becomes easier because the evaluator can validate:

schema correctness
allowed values
required fields
consistency between fields
tool validity

This is considerably easier to evaluate than unconstrained natural language.

15. Measuring Agent Quality

Agent evaluation should go beyond final-answer accuracy.

Useful metrics include:

Accuracy

Did the agent provide the correct answer?

Tool-call accuracy

Did the agent call the correct tools?

Routing accuracy

Did it traverse the correct graph path?

Retrieval quality

Did it retrieve relevant information?

Reliability

Does it behave consistently across repeated runs?

Efficiency

How many LLM calls and tokens were required?

Latency

How long did the full execution take?

This allows an agent to be evaluated as a software system rather than merely as a chatbot.

16. Regression Testing for Agents

Every production failure can become a future test case.

For example:

Production incident
        ↓
Identify failure mode
        ↓
Create regression scenario
        ↓
Add to evaluation dataset
        ↓
Prevent recurrence

Over time, the scenario dataset becomes a practical representation of the application's business requirements.

This is particularly valuable because agent behaviour can change significantly when:

prompts change
models change
tools change
graph logic changes
retrieval indexes change

17. Building an Agent Evaluation Dataset

A useful evaluation dataset should contain:

Field	Purpose
User question	Original scenario
Conversation context	Relevant history
Expected intent	Required interpretation
Expected tools	Valid tool usage
Expected entities	Project/property/etc.
Expected outcome	Required behaviour
Failure conditions	Unacceptable behaviour
Evaluation result	Pass/fail or score

This makes agent testing repeatable rather than dependent on manually inspecting conversations.

18. Mocking vs Real Dependencies

Different tests require different levels of realism.

For unit tests, mock external dependencies aggressively.

For integration tests, use real databases and real tools where practical.

For end-to-end evaluation, test the complete production-like pipeline.

A useful principle is:

The lower the test level, the more deterministic it should be.

The higher the test level, the more realistic it should be.

19. Controlling LLM Variability During Testing

LLM behaviour introduces variance.

For stable evaluation, control variables such as:

model version
temperature
system prompt
tool definitions
retrieval dataset
structured output schema

Even then, exact output matching is usually inappropriate.

Evaluation should focus on semantic correctness and required behaviour.

20. Common Testing Mistakes

Several approaches tend to fail in production.

Testing everything through the LLM

This is expensive and makes failures difficult to diagnose.

Testing only individual functions

This misses orchestration and graph-level failures.

Using only exact answer matching

A correct answer can be expressed in many valid ways.

Testing only happy paths

Production failures frequently occur in ambiguous and incomplete requests.

Ignoring tool-call traces

The final answer may look reasonable even though the agent reached it through an invalid process.

21. A Practical Testing Architecture

A practical architecture can therefore look like:

                ┌──────────────────────┐
                │  Unit Tests          │
                │  Deterministic Code  │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Tool Tests           │
                │  DB / Retrieval       │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Graph Tests          │
                │  Routing / State      │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Agent Evaluation     │
                │  Scenario Dataset     │
                └──────────┬───────────┘
                           │
                ┌──────────▼───────────┐
                │  Production Feedback  │
                │  Regression Cases     │
                └──────────────────────┘

The important point is that no single testing strategy is sufficient.

22. Lessons Learned from Production Agent Development

The main lesson is that testing an agent is fundamentally an exercise in decomposition.

Trying to test the entire system through end-to-end LLM calls is expensive.

Trying to test the entire system through traditional unit tests is insufficient.

The more practical approach is to divide the system into deterministic and probabilistic boundaries.

Deterministic logic should be tested directly.

Tools should be tested independently.

Graph execution should be tested with controlled inputs.

LLM reasoning should be evaluated through realistic scenarios.

End-to-end tests should verify whether the complete system satisfies business requirements.

23. Conclusion

Production AI agents sit somewhere between software engineering and probabilistic systems engineering.

The challenge is not simply making an LLM produce a good answer.

The challenge is making the entire system predictable enough to operate reliably:

LLM + tools + graph + retrieval + business logic + state

That changes how testing needs to be designed.

Rather than asking:

"Can I unit test this agent?"

A better question is:

"Which parts of this agent are deterministic, which parts are probabilistic, and what is the correct testing strategy for each?"

That distinction has become one of the most useful principles in building stable production agents.

Learning OCR Process from Scratch – 004: Parsing OCR Detection Output

kai wen ng — Sun, 23 Aug 2026 05:08:42 +0000

In the previous post, I ran the PP-OCRv4 detection model and obtained its raw probability map.
But this probability map is not yet a list of text boxes.
Now we need to turn that prediction map into actual detection regions.

The basic flow is:
Probability Map → Thresholding → Contours → Box Filtering → Box Expansion → Cropped Text Region

1. Filtering Low-Confidence Regions

The first step is to convert the probability map into a binary image.

cv::Mat bitmap;
cv::threshold(pred,bitmap,thresh,255,cv::THRESH_BINARY);

Here, thresh is the threshold used to separate likely text regions from the background.

Pixels above the threshold become 255, while the remaining pixels become 0.

This gives us a binary mask of the regions where the model believes text exists.

2. Finding the Contours

The binary image can now be used to find connected regions:

bitmap.convertTo(bitmap, CV_8U);
std::vector<std::vector<cv::Point>> contours;
cv::findContours(bitmap * 255,contours,cv::RETR_LIST,cv::CHAIN_APPROX_SIMPLE);

The contour representation is:

vector<vector<Point>>

Each vector<Point> represents one contour, while each Point represents a coordinate along that contour.

There are two parameters here that are important.

cv::RETR_LIST tells OpenCV to retrieve all contours without building a parent-child hierarchy between them.

cv::CHAIN_APPROX_SIMPLE compresses the contour representation by removing redundant points. For example, points along a straight line do not all need to be stored.

At this point, the probability map has been transformed into a collection of geometric regions.

3. Filtering Small and Low-Confidence Regions

Not every detected region should become a text box.

I first filter out regions that are too small, such as contours with a width or height below 3 pixels.

I also calculate the confidence score of each region and discard boxes with a score below my threshold of 0.7.

This helps remove noise and tiny regions that are unlikely to represent actual text.

4. Ordering the Box Points

After extracting a rectangular detection region, I need to make sure its four corner points are in a consistent order.
The points are first sorted by their x-coordinate:

std::sort(
    boxPoints.begin(),
    boxPoints.end(),
    [](const cv::Point2f &a, const cv::Point2f &b) {
        return a.x < b.x;
    }
);

From there, I determine which points correspond to the top-left, top-right, bottom-left, and bottom-right corners. Which is important for detection regions to be cropped later on.

std::vector<cv::Point2f> box = {
    boxPoints[index1],
    boxPoints[index2],
    boxPoints[index3],
    boxPoints[index4]
};

5. Calculating the Detection Confidence

The contour itself does not directly tell me how confident the model is about the detected region.

Instead, I calculate the average probability inside the detected polygon.

First, I create a mask and fill the detected polygon:

cv::fillPoly(
    mask,
    std::vector<std::vector<cv::Point>>{polygon},
    cv::Scalar(255)
);

Then I calculate the mean value of the original probability map inside that mask:

float score = static_cast<float>(cv::mean(pred, mask)[0]);
if (score < boxThresh)
{continue;}

This gives me an average confidence score for the detected region.

If the score is too low, the detection is discarded.

This is different from the thresholding step earlier.

The first threshold creates candidate regions from the probability map. This second score checks how confident the model is about the entire candidate region.

6. Expanding the Detection Box

The detected contour usually does not perfectly cover the complete text.
It can be slightly smaller than the actual text region, so RapidOCR expands the box before cropping it.
The expansion distance is calculated from the rectangle's area and perimeter:

float area = rect.size.width * rect.size.height;

float perimeter = 2.0f * (rect.size.width +rect.size.height);

float distance = area * unclip_ratio / perimeter;

The rectangle is then expanded:

rect.size.width += 2.0f * distance;
rect.size.height += 2.0f * distance;

Finally, the expanded rectangle is converted back into four cornerpoints:

cv::Point2f expandedPoints[4];

rect.points(expandedPoints);

std::vector<cv::Point2f> expandedBoxPoints(
    expandedPoints,
    expandedPoints + 4
);

The points are then sorted again into a consistent order.

This expansion is useful because a detection that is too tightly cropped can cut off parts of the characters, which can negatively affect the recognition model.

7. From Detection to Recognition

At this point, the detection stage is essentially complete.
These cropped regions can now be passed to the next stages of the OCR pipeline, such as orientation classification and text recognition.

What I Learned

This was another step where I realised that the neural network is only part of the OCR system.

The detection model gives us a probability map, but it does not directly give us the final bounding boxes.

The rest is classical computer vision and geometry processing:

Thresholding → Contour Extraction → Filtering → Polygon Scoring → Box Ordering → Box Expansion

So the OCR detection stage is really a combination of deep learning and traditional computer vision.

The next step is to take these cropped regions and understand how the recognition model turns them into actual characters.

Learning OCR Process from Scratch – 004

Learning OCR Process from Scratch – 002: From Image to ONNX Tensor

kai wen ng — Sat, 22 Aug 2026 23:51:22 +0000

In the previous post, I explored the basic OCR workflow and discovered that OCR is not simply a single model inference call.

There are multiple stages involved, from text detection and orientation classification to text recognition and decoding.

This time, I wanted to go one level deeper:

What actually happens to the image before it reaches the OCR model?

1. Loading the Image

Since I am implementing the inference pipeline in C++, the first step is loading the image into memory using OpenCV:

cv::Mat image = cv::imread(filename, cv::IMREAD_COLOR);

At this point, we simply have an OpenCV cv::Mat.
However, the ONNX model cannot directly consume this object.
We need to transform the image into the exact tensor representation expected by the model.

2. Resizing the Image

Before inference, the image dimensions also need to be checked.
For the detection model, images that are too large or too small need to be resized according to the preprocessing logic used by the original OCR pipeline.
This is important because the model is trained within a particular input distribution. Feeding images with dimensions that are significantly different from what the preprocessing pipeline expects can affect detection accuracy.
So the image first goes through the appropriate resizing step before being passed to the model.

3. Converting and Normalising the Image

Next, the image needs to be converted from its original data type into float32.
The pixel values are also normalised by dividing them by 255:

cv::Mat floatImage;
image.convertTo(floatImage, CV_32F, 1.0 / 255);

Now the pixel values are represented as floating-point numbers in approximately the range:

[0, 1]

But there is another problem.
The image is still represented as an OpenCV cv::Mat, while the ONNX model expects a tensor.

4. Understanding the ONNX Input

I inspected the detection model using netron:

ch_PP-OCRv4_det_infer.onnx

and found that its input shape is:

[-1, 3, -1, -1]

The -1 dimensions are dynamic, while the 3 represents the colour channels.
For inference, I provide the concrete shape:

[1, 3, H, W]

This follows the common NCHW tensor layout:

N = batch
C = channels
H = height
W = width

So I need to transform the OpenCV image from its image representation into this tensor layout.

5. Converting HWC to CHW

OpenCV stores the image spatially as:

H × W × C

But the ONNX model expects:

C × H × W

Therefore, the channels need to be separated first:

std::vector<cv::Mat> channels;
cv::split(floatImage, channels);

I then allocate a contiguous block of memory for the tensor:

std::vector<float> inputTensorValues(3 * srcH * srcW);

Instead of copying every pixel individually, I can copy each channel directly into the appropriate region of the tensor using std::memcpy and ptr<float>(), which gives me a pointer to the first float element of a specified row. When called without a row index, it points to the first element of the matrix.

std::memcpy(
    inputTensorValues.data(),
    channels[0].ptr<float>(),
    srcH * srcW * sizeof(float)
);

std::memcpy(
    inputTensorValues.data() + srcH * srcW,
    channels[1].ptr<float>(),
    srcH * srcW * sizeof(float)
);

std::memcpy(
    inputTensorValues.data() + 2 * srcH * srcW,
    channels[2].ptr<float>(),
    srcH * srcW * sizeof(float)
);

The offsets are important here.
Each channel contains:

H × W

floating-point values.
Therefore:

Channel 0 starts at offset 0
Channel 1 starts at offset H × W
Channel 2 starts at offset 2 × H × W This gives us a contiguous CHW representation in memory.

6. Defining the Tensor Shape

The data itself is not enough.
ONNX Runtime also needs to know how that block of memory should be interpreted.
So I define the tensor shape:

std::vector<int64_t> inputShape = {1, 3, srcH, srcW};

The shape tells the runtime that the memory represents:

Batch × Channel × Height × Width

7. Creating the ONNX Runtime Tensor

Finally, I need to tell ONNX Runtime how the memory should be handled.
First, I create the memory information:

Ort::MemoryInfo memoryInfo(
    "Cpu",
    OrtDeviceAllocator,
    0,
    OrtMemTypeDefault
);

Then I create the actual tensor:

Ort::Value inputTensor = Ort::Value::CreateTensor<float>(
    memoryInfo,
    inputTensorValues.data(),
    inputTensorValues.size(),
    inputShape.data(),
    inputShape.size()
);

At this point, the original image has gone through the following transformation:

The Interesting Part

What surprised me here was how much work is hidden behind a simple Python inference call.

In Python, the framework handles most of this conversion for me.

In C++, I have to explicitly deal with:

Image preprocessing
Data types
Tensor shapes
Channel ordering
Memory layout
Memory allocation
Tensor construction

This is also where I started to appreciate the difference between running a model and building an inference pipeline.

The model itself is only one part of the system.

The preprocessing and memory representation have to match what the model expects as well.

In the next step, I can finally pass this tensor into ONNX Runtime and start looking at what actually comes out of the detection model.

Learning OCR Process from Scratch – 002

SQLAlchemy One-to-Many Relationships

kai wen ng — Tue, 18 Aug 2026 13:29:41 +0000

Suppose we have:

One Project
Many Property So the relationship is:

Project 1 ────────< Property

The many side (Property) owns the foreign key pointing to the primary key of the one side (Project).

1. Foreign Key

For example:

class Project(Base):
    __tablename__ = "project"

    id = mapped_column(primary_key=True)


class Property(Base):
    __tablename__ = "property"

    id = mapped_column(primary_key=True)
    project_id = mapped_column(ForeignKey("project.id"))

The database structure is conceptually:

project
-------
id (PK)
1
2
3


property
--------
id (PK)
project_id (FK)
101   1
102   1
103   1
104   2

Here:

Project 1 has properties 101, 102, and 103
Project 2 has property 104

So:

Project 1 → many Property
Property 101 → one Project

The foreign key is usually named something like:

project_id

because it refers to project.id.

2. relationship()

The ForeignKey establishes the database-level relationship.
The relationship() establishes the ORM-level relationship.

For example:

class Project(Base):
    __tablename__ = "project"

    id = mapped_column(primary_key=True)

    properties = relationship(
        "Property",
        back_populates="project"
    )


class Property(Base):
    __tablename__ = "property"

    id = mapped_column(primary_key=True)
    project_id = mapped_column(ForeignKey("project.id"))

    project = relationship(
        "Project",
        back_populates="properties"
    )

Now SQLAlchemy lets you navigate the relationship through Python attributes:

project.properties

returns:

[Property(...), Property(...), Property(...)]

while:

property.project

returns:

Project(...)

So the type relationship is:

Project.properties → list[Property]
Property.project   → Project

`back_populates`

back_populates tells SQLAlchemy that these two relationships are two sides of the same relationship.

Project.properties
        ↕
Property.project

Because of this, SQLAlchemy can keep the in-memory object graph synchronised.

For example:

project = Project()
property = Property()

project.properties.append(property)

and SQLAlchemy understands:

project.properties
        ↓
property.project = project
        ↓
property.project_id = project.id

So when the project gets its primary key and the session flushes, SQLAlchemy can insert the corresponding project_id into property.

Important distinction

back_populates does not create the foreign key.
These have separate responsibilities:

ForeignKey     → database relationship
relationship() → ORM relationship
back_populates → links the two ORM relationship attributes

3. Lazy Loading

By default, SQLAlchemy relationships are generally loaded lazily.
Suppose:

project = session.get(Project, 1)

SQLAlchemy initially retrieves the project:

SELECT *
FROM project
WHERE id = 1;

The properties have not necessarily been loaded yet.
When you do:

project.properties

SQLAlchemy may then issue another query:

SELECT *
FROM property
WHERE project_id = 1;

So:

Load Project
     ↓
Access project.properties
     ↓
Load Properties

This is useful because you don't retrieve related data unnecessarily.
However, repeatedly doing this can cause the N+1 query problem.
For example:

projects = session.query(Project).all()

for project in projects:
    print(project.properties)

Conceptually this can result in:

1 query → load all projects
N queries → load properties for each project

So:

1 + N queries

4. Eager Loading

Eager loading tells SQLAlchemy to retrieve the related objects as part of the initial loading process.
For example:

from sqlalchemy.orm import selectinload

projects = session.scalars(
    select(Project).options(
        selectinload(Project.properties)
    )
).all()

SQLAlchemy can then load the projects and their properties using additional planned queries, typically something like:

SELECT *
FROM project;

followed by:

SELECT *
FROM property
WHERE project_id IN (...);

So instead of:

1 + N queries

you can often get:

2 queries

The main difference are:

Lazy loading  → related data loaded when accessed
Eager loading → related data loaded as part of the intended fetch operation

Memory trade-off

Eager loading can increase memory usage because more ORM objects are materialised immediately.
However, the performance trade-off is more nuanced than simply:

Lazy   = more queries
Eager  = fewer queries

The best strategy depends on:

number of related rows
query frequency
result-set size
whether relationships are actually needed
type of eager loading (joinedload, selectinload, etc.)

5. Cascade

This part of your notes needs the most correction.
Cascade controls what ORM operations on a parent are propagated to related objects.
For example:

properties = relationship(
    "Property",
    back_populates="project",
    cascade="all, delete-orphan"
)

delete-orphan adds another rule:
A child that is no longer associated with its parent is considered an orphan and will be deleted.

all is effectively a shorthand for several cascade behaviours, including:

save-update
merge
refresh-expire
expunge
delete

More on that at: https://docs.sqlalchemy.org/en/21/orm/cascades.html

SQLAlchemy's default save-update cascade generally causes related transient objects to be pulled into the session when they are associated through the relationship.

For example:

project = Project()

property = Property()

project.properties.append(property)

session.add(project)

Because of the relationship cascade, the property can also become part of the session.
Then:

session.commit()

can persist both objects.
Conceptually:

session.add(project)
       ↓
Project tracked
       ↓
Property associated through relationship
       ↓
Property also tracked
       ↓
INSERT Project
INSERT Property

Overall...

A useful way to remember the four concepts is:
| Concept | Responsibility |
| ---------------- | -------------------------------------------- |
| ForeignKey | Defines the relationship in the database |
| relationship() | Defines the relationship in the ORM |
| back_populates | Connects the two ORM sides |
| cascade | Defines how certain ORM operations propagate |
| Lazy loading | Load relationship when accessed |
| Eager loading | Load relationship as part of the query |

One additional distinction worth remembering is that ORM cascade and database-level ON DELETE CASCADE are different mechanisms. SQLAlchemy can delete child objects itself, while the database can also be configured to perform cascading deletes via the foreign key.

So the overall structure is:

                 DATABASE
                     │
            ForeignKey(project_id)
                     │
                     ▼
Project 1 ─────────< Property
   │                    │
   │ relationship()     │ relationship()
   │                    │
   ▼                    ▼
properties            project
   │                    │
   └──── back_populates┘
            │
            ▼
         cascade
            │
            ▼
   controls ORM operation
       propagation

Useful Reading & References:
https://learn.miguelgrinberg.com/product/sqlalchemy2

8 Things I learnt from JavaScript First Time Developing Frontend

kai wen ng — Sun, 16 Aug 2026 05:40:52 +0000

1. The DOM is the interface between JavaScript and HTML

DOM (Document Object Model) is the programming interface provided by web browsers that allows JavaScript to read and modify the HTML page dynamically.

document → represents the web page.
Object → represents HTML elements/tags.
Model → represents how elements are structured and nested, essentially forming an HTML tree.

Finding elements

querySelector() locates an element using a CSS selector. It returns the first matching element.
closest() finds the nearest ancestor that matches a selector.

let input = document.querySelector("input[name='action']");
let row = btn.closest("tr");

Given a button, its containing form can also be accessed directly:

const form = btn.form;

This is useful when an action needs to modify or submit related form data.

2. JavaScript can dynamically generate HTML

Instead of defining every element statically in HTML, JavaScript can create UI elements dynamically based on application data.

const input = document.createElement("input");
input.value = attachment.description;

This is useful for dynamically generated forms, tables, buttons, inputs, and other UI components.

3. Event handlers connect user actions to application logic

Events provide the bridge between user interaction and JavaScript logic.
Common events include:

click
change
input
submit

For example:

input.addEventListener("change", updateData);

When the input changes, JavaScript executes updateData().

Passing the triggering element

An inline event handler can pass the actual HTML element to a function:

onchange="updateData(this)"

Then:

function updateData(input) {
    let row = input.closest("tr");

    if (!row || !row.data) {
        return;
    }

    row.style.backgroundColor = "#fc5e5e";
}

Here, this refers to the element that triggered the event.
This allows the function to locate related elements and modify their UI or associated data.

## 4. JavaScript looping over objects and collections

JavaScript provides different ways to iterate over data depending on whether you need object keys, values, or both.

`Object.entries()` — loop over object key-value pairs

Object.entries() converts an object into an array of [key, value] pairs.
It is conceptually similar to Python's dict.items().

const data = {
    name: "abc",
    category: "image"
};

for (const [key, value] of Object.entries(data)) {
    console.log(key, value);
}

Output:

name abc
category image

This is useful when you need both the property name and its value.

`for...of` — loop over values

for...of is useful when you only need the values in an iterable such as an array.

for (const categoryType of categoryTypes) {
    // process categoryType
}

Instead of:

for (let i = 0; i < categoryTypes.length; i++) {
    const categoryType = categoryTypes[i];
}

5. JavaScript can modify form values programmatically

JavaScript can directly modify the value of an HTML form field:

document.querySelector("input[name='action']").value = "XXX";

This allows the frontend to add or modify state before a form is submitted.
For example, the UI can set an action value based on which button the user clicked, allowing the backend to determine what operation should be performed.

6. DataTables provides an interactive JavaScript table

DataTables is a JavaScript library that transforms a static HTML <table> into an interactive data grid.
It provides features such as:

Sorting
Searching
Pagination
Dynamic row manipulation

Example:

let attachmentTable = $('#table').DataTable();

Attaching application data to a DataTables row

A row can be added dynamically, and its corresponding HTML <tr> DOM element can be retrieved using .node():

let row = table.row.add([
    col1_data,
    col2_data,
    col3_data,
    col4_data
]).draw(false).node();
row.data = data;

This creates a useful connection between:

Application data
      ↓
DataTables row
      ↓
HTML <tr> element

The custom row.data property can then be used to associate the underlying application object with the visible UI row.

8. `Promise.all()` runs asynchronous operations concurrently

When multiple requests are independent, they can be started concurrently and waited for together.

requests.push(fetch(...));

Then:

await Promise.all(requests);

Instead of:

await request1;
await request2;
await request3;

which executes them sequentially, Promise.all() allows the requests to run concurrently.
Conceptually:

Request 1 ────────────────┐
Request 2 ───────────┐   │
Request 3 ────────────┐  │
Request 4 ────────────┤  │
                      ↓  ↓
                 Promise.all()

This is similar to Python's:

await asyncio.gather(*tasks)

10. The bigger lesson: understand the data flow, not just the syntax

When reading unfamiliar or legacy JavaScript, focus on the data flow rather than trying to understand every line independently.
Ask these questions:

Where does the data come from?
        ↓
How is the data transformed?
        ↓
Where is the data stored?
        ↓
What UI changes?
        ↓
What API/backend action happens next?

This helps reconstruct the application's execution model.

For example:

User clicks button
      ↓
Event handler executes
      ↓
Find related DOM element
      ↓
Retrieve application data
      ↓
Modify data / UI
      ↓
Send request to backend
      ↓
Receive response
      ↓
Update UI

The important skill is therefore not memorising JavaScript syntax, but understanding how data moves between the DOM, JavaScript state, UI components, and the backend.

Building an Inference Engine from Scratch - Part 2: Basics of Attention Mechanism

kai wen ng — Wed, 12 Aug 2026 04:30:00 +0000

Before building an LLM inference engine, I need to understand the fine-grained details behind how modern LLMs work.

The book "Build a Large Language Model from Scratch" has been an excellent companion throughout this journey:

https://www.manning.com/books/build-a-large-language-model-from-scratch

It not only explains the theory behind Transformers, but also provides a practical understanding of how each component connects together to form a working language model.

Why Transformers?

Before Transformers, many sequence models were based on RNN architectures.
However, RNNs have several limitations:

Sequential computation makes training and inference slow.
Long sequences suffer from information loss due to limited hidden-state memory.
Parallelization is difficult.

What is Attention?

At its core, attention allows each token to determine which other tokens are important when building its representation.
Starting from token embeddings:

Token Embeddings
        |
        |
Linear Projection
        |
   --------------
   |     |      |
   Q     K      V

Each embedding is transformed into:

Query (Q): What information is this token looking for?
Key (K): What information does this token contain?
Value (V): What information should be passed forward?

For each token:

The query vector is compared with all key vectors using dot product:

Attention Score = Q × Kᵀ

The scores are normalized using softmax:

Attention Weight = Softmax(Attention Score)

The weights are applied to the value vectors:

Context Vector = Attention Weight × V

The result is a context-aware representation where each token contains information from other relevant tokens.

Multi-Head Attention

Instead of learning only one attention pattern, Transformers use multiple attention heads.
Each head can learn different relationships:

Syntax relationships
Long-distance dependencies
Semantic similarity
Position-based patterns

The outputs from all heads are combined to create a richer representation.

Causal Attention

For autoregressive language models, the model should only use previous tokens when predicting the next token.

Example:

The cat sat on the ...

When predicting the next word, the model should not have access to future tokens.
This is achieved using a causal mask:

Before Softmax:

[0    -inf  -inf]
[0     0    -inf]
[0     0     0 ]

The -inf values become zero probability after applying softmax.

Dropout

Dropout is a regularization technique used during training. During training, some activations are randomly disabled so the model does not depend too heavily on specific neurons or pathways. This encourages the network to learn more robust representations and reduces overfitting. During inference, dropout is disabled.

Layer Normalization

Layer normalization stabilizes activations throughout the network. It normalizes hidden representations and applies learnable:

Scale parameters
Shift parameters

This helps maintain stable signal propagation through deep Transformer layers.

Activation Function: GELU

Modern Transformers commonly use GELU instead of ReLU.
ReLU:

ReLU(x) = max(0, x)

completely removes negative values.
GELU provides a smoother activation function, allowing small negative inputs to contribute instead of being completely discarded which in turn improved representation learning in deep networks.

Hidden Dimension / Neurons

The hidden dimension determines the size of the representation space.
A larger hidden dimension allows the model to capture more complex patterns and relationships.

However:

More parameters require more memory.
More computation is required.
Larger models require more training data.

Residual Connections (Shortcut Connections)

Transformer blocks contain residual connections:

Output = Layer(x) + x

These shortcut paths:

Improve gradient flow.
Allow deeper networks to train effectively.
Preserve information between layers.

This concept was introduced in residual networks and became a fundamental component of modern deep learning architectures.

Transformer Block

A Transformer is not only an attention mechanism.
A typical Transformer block contains:

Input
 |
Layer Normalization
 |
Multi-Head Attention
 |
Residual Connection
 |
Layer Normalization
 |
Feed Forward Network
 |
Residual Connection
 |
Output

Keeping input and output dimensions consistent allows multiple Transformer blocks to be stacked together.

Model Parameters

To understand the size of an LLM, we need to know how many trainable parameters it contains.
During model implementation, the number of parameters can be calculated using numel, which counts the total number of elements inside a tensor.
For example, each weight matrix and bias vector contributes to the overall parameter count:

Total Parameters = numel(all model tensors)

Logits and Token Generation

The final layer outputs logits.
Logits are raw scores before converting them into probabilities.

Applying softmax:

Logits → Softmax → Probability Distribution

gives the probability of each possible next token.
The simplest decoding method:

argmax(probability)

selects the highest probability token.
To generate more diverse outputs, modern LLMs often use sampling strategies on the probability distribution

Top-k sampling
Top-p sampling

Autoregressive Generation Loop

LLMs generate text one token at a time.

The process:

Input tokens are passed through the Transformer.
The model predicts the next token.
The predicted token is appended to the input.
The process repeats. This simple loop is the foundation behind text generation in modern LLMs.

Building an inference engine from scratch is giving me a deeper understanding of what happens behind every API call.
The goal is not only to run models, but to understand every matrix multiplication, tensor transformation, and design decision that makes modern LLMs possible.

Speeding Up Vector Search with Quantization & Vector Indices

kai wen ng — Tue, 11 Aug 2026 04:30:00 +0000

Unlike traditional databases, vector databases do not search for exact matches. Instead, they compare the similarity between high-dimensional embedding vectors. A naive implementation would compare the query vector against every vector in the database, which quickly becomes computationally expensive as the dataset grows.

To improve search speed, vector databases use specialised indexing algorithms that significantly reduce the number of vector comparisons while maintaining high retrieval accuracy.

Flat Index

The Flat index is the simplest approach. It performs a brute-force search by calculating the similarity between the query vector and every vector stored in the database.

IVF (Inverted File Index)

IVF speeds up vector search by first clustering the dataset using algorithms such as K-Means.
Instead of searching the entire database, the query is first assigned to the nearest cluster, and only vectors within that cluster (or a few nearby clusters) are searched.
Conceptually, it behaves like an inverted index which tells the search: this cluster has this vector

Cluster A
 ├── Vector 1
 ├── Vector 5
 └── Vector 12

Cluster B
 ├── Vector 2
 ├── Vector 7
 └── Vector 20

LSH (Locality-Sensitive Hashing)

Locality-Sensitive Hashing maps similar vectors into the same hash buckets with high probability.
Unlike traditional hash functions that minimise collisions, LSH intentionally increases collisions for similar vectors. It does this by projecting vectors onto multiple random hyperplanes and generating hash codes based on the projection results.
During retrieval, only vectors in the same hash buckets are compared instead of the entire database.

HNSW (Hierarchical Navigable Small World)

HNSW is one of the most popular vector indexing algorithms today because it offers an excellent balance between speed and accuracy.
It organises vectors into a hierarchical graph.

The upper layers contain only a small number of nodes connected by long-distance links, similar to highways connecting major cities.
Lower layers contain many more nodes with shorter connections, similar to local roads.

A search begins from the top layer and gradually moves down through increasingly detailed layers until it reaches the nearest neighbours.

Important parameters include:

M – Maximum number of roads connected to each city. More roads create a denser network, making it easier to find efficient routes.
efConstruction – Determines how thoroughly the road network is planned during construction. Higher values spend more time finding the best roads between cities, resulting in a better network.
efSearch – Determines how many possible routes are explored during navigation. Higher values consider more roads before deciding on the destination, improving accuracy at the cost of slightly longer search time.

Quantization

Even with efficient indexes, storing and comparing high-dimensional floating-point vectors remains expensive.
Quantization reduces both memory usage and computation by compressing vectors before indexing.

Scalar Quantization (SQ)

Scalar Quantization compresses each floating-point value individually.
For example:

Original: 32-bit floating-point numbers
Compressed: 8-bit integers

Product Quantization (PQ)

Product Quantization compresses vectors more aggressively.
Instead of compressing each value independently, it:

Splits each embedding into multiple smaller sub-vectors.
Runs clustering on each sub-vector independently.
Stores only the centroid IDs instead of the original floating-point values.

During retrieval:

The query embedding is split into the same sub-vectors.
Distances are computed using the centroid codebooks instead of the original vectors.

Additional Optimisations

Vector search is rarely used alone in production systems. Several additional techniques are commonly combined to improve both performance and retrieval quality.

Metadata Filtering

Before performing vector search, metadata can eliminate documents that are impossible matches.
Examples include:

File type
Language
Date range
User permissions
Product category

Reducing the candidate set before vector search significantly improves efficiency.

Hybrid Search

Many production retrieval systems combine vector search with traditional keyword search.
Keyword search provides exact lexical matching, while vector search captures semantic similarity.
The results are then combined using ranking algorithms, often producing higher-quality retrieval than either approach alone.

Reference: https://www.oreilly.com/library/view/vector-databases/9781098177584/

How Maple Tree Manages VMAs

kai wen ng — Mon, 10 Aug 2026 04:30:00 +0000

Starting from Linux 6.1, the kernel replaced both the linked list (mmap) and the red-black tree (mm_rb) used to manage Virtual Memory Areas (VMAs) with a new data structure called the Maple Tree.
Maple Tree is a range-based tree structure designed for storing non-overlapping ranges.
It provides:

Fast lookup
Efficient traversal
Better memory usage
Read Copy and Update (RCU) safe operations Unlike a red-black tree, Maple Tree stores ranges directly, which matches how VMAs are represented.

Maple Tree Structure

Maple Tree nodes contain:

Slots
Pivots Each node 15x pivots and 16x entries. A slot stores an entry, while pivots define the range boundaries. The range information is implied by the pivot values instead of storing every range boundary explicitly. This allows Maple Tree to store large numbers of memory ranges efficiently.

Replace Operation

When modifying a Maple Tree node:

Maple Tree locates the node containing the target range.
A new empty node is allocated.
Existing entries and pivots before the modified range are copied into the new node.
The node is split at the position where the new range will be inserted.
The new entry is inserted into the correct slot.
The remaining entries and pivots after the modified range are copied into the new node.
The original node is replaced with the newly created node.

This replacement approach allows Maple Tree to support safe concurrent reads using RCU. Readers can continue accessing the old node while the writer prepares the updated version. Once the new node is complete, the kernel switches the reference to the new node.

Node Splitting

When a Maple Tree node becomes full and cannot store another entry:

The full node is removed from the active tree structure.
Two new nodes are created.
Existing entries and pivots are redistributed between the two new nodes.
The new entry is inserted into the correct position during the redistribution process.
The parent node is updated with the new child nodes and new pivot values. The split operation tries to keep the data density balanced between the two nodes while avoiding unnecessary tree height growth. After splitting, the parent node receives an additional slot pointing to the new node. The new pivot information is updated to represent the new range boundaries.

Maple Tree tries to maintain:

High Data Density: Nodes should contain many entries to reduce memory usage and improve cache efficiency.
Lower Tree Height: A shorter tree reduces the number of steps required during lookup.
Avoid Underfilled Nodes: After deletion or modification, entries may be redistributed or merged to avoid wasting space.

When multiple entries can be represented more efficiently as a single range:

A new node is created for the reorganized entries.
Existing entries are copied from the previous nodes.
Entries that can be represented as a continuous range are compressed into a single slot and pivot.
Remaining entries are moved together with neighbouring nodes.
The parent node is updated with the new node layout and pivot information.

This reduces the number of slots required and keeps the tree compact.
The goal of rebalancing is to maintain efficient storage by reducing unnecessary nodes while keeping lookup performance high.

Special thanks to the Linux Foundation and Liam Howlett's talk "The Maple Tree: Structure and Algorithms", which helped me understand the internal design and algorithms behind Maple Tree.
Link: https://www.youtube.com/watch?v=RaXhP-QLUxI

Towards the Stability of LLM Agents

kai wen ng — Sun, 09 Aug 2026 09:02:47 +0000

An industry-level LLM agent is not simply an API call that returns a response. It needs to be resilient to transient failures, malformed outputs, and schema violations.
To improve the stability of my agent system, I introduced two decorators around my LLM calls. They handle three common failure modes:

JSON decoding errors — provide the model with its previous malformed response and ask it to regenerate valid JSON.
Pydantic BaseModel validation errors — provide the model with the specific schema validation errors and ask it to correct the response.
429 rate-limit errors — retry with appropriate backoff.

This separates the reliability logic from the individual agent functions, allowing each LLM call to focus on its own task while the decorators provide a consistent recovery mechanism.

Json and Pydantic Basemodel

To handle problematic model outputs, I first use the json-repair package:
json-repair
PS: If you find the package useful, consider giving the author a ⭐ to show some support.

This allows the system to recover from common JSON formatting issues in the model's response. If the repaired result is still empty, I raise a JSONDecodeError manually.

This is useful because the retry decorator can then intercept the exception and provide the model with a meaningful retry message. Instead of blindly asking the model to try again, I can tell it:

Your previous response was not valid JSON.

and include the previous response through e.doc. This gives the model its failed output as context and allows it to regenerate the response correctly.

The next case is Pydantic ValidationError. A response can be valid JSON but still fail to match the expected schema. In this case, simply asking the model to retry is also not particularly useful. The retry message should contain the specific fields that failed validation and the expected constraints.

This information is available through e.errors(). I can extract the field location and validation message, then feed those details back to the model. This turns the retry from a blind regeneration into a targeted correction.


def llm_retry(max_retries=3):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            retry_message = None

            for attempt in range(max_retries):
                try:
                    return await func(*args, retry_message=retry_message, **kwargs)

                except ValidationError as e:
                    errors = [
                        {"field": ".".join(map(str, x["loc"])), "message": x["msg"]}
                        for x in e.errors()
                    ]
                    retry_message = (
                        "Your previous response failed schema validation.\n\n"
                        f"Validation errors:\n{errors}\n\n"
                        "Correct these errors and return the complete response as JSON only."
                    )

                except json.JSONDecodeError as e:
                    retry_message = (
                        "Your previous response was not valid JSON.\n\n"
                        f"Previous response:\n{e.doc}\n\n"
                        "Regenerate the complete response as valid JSON only. "
                        "Do not include Markdown or explanations."
                    )

            raise RuntimeError(f"LLM failed after {max_retries} attempts")

        return wrapper
    return decorator

The 429 Rate-Limit Decorator

This part is relatively straightforward. I use a simple function factory to create a decorator that wraps around the LLM calls and automatically retries requests that receive a 429 Too Many Requests response.

The decorator applies an exponential backoff between attempts, allowing the system to recover from temporary rate limiting without requiring each LLM call to implement its own retry logic.

def Retry429(trials=3, delay=1.0, backoff=2.0):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            current_delay = delay

            for attempt in range(trials):
                try:
                    return await func(*args, **kwargs)
                except RateLimitError:
                    pass
                except httpx.HTTPStatusError as e:
                    if e.response.status_code != 429:
                        raise
                except httpx.HTTPError:
                    raise

                if attempt == trials - 1:
                    raise

                await asyncio.sleep(current_delay)
                current_delay *= backoff

        return wrapper

    return decorator

After having these decorators in place, I can define an LLM call as a simple function. Each function is responsible only for constructing the input arguments and returning the output arguments, represented by LLMRequest and LLMResponse.

Defining these interfaces as Pydantic BaseModels provides a consistent contract between different parts of the system and makes the overall agentic architecture more systematic and maintainable.

More importantly, the retry and recovery mechanisms are no longer coupled to individual LLM implementations. This allows me to support different types of LLM calls, such as image generation and VLM inference, without copy-pasting the same retry and validation logic across every function. The individual functions remain focused on their specific task, while the shared decorators handle the common reliability concerns.


    @Retry429(trials=10, delay=10)
    @llm_retry()
    async def generate(self, llm_request: LLMRequest, retry_message=None)-> LLMResponse:
        args = {}
        # Just some args parsing
        if retry_message is not None:
            args["messages"].append({"role": "system", "content": retry_message})

        response = await self.client.chat.completions.create(**args)
        self.add_token_usage(response)
        if not response:
            raise RuntimeError("LLM returned empty response")

        if not response.choices:
            logger.error(
                "LLM returned no choices. Response=%s",
                response.model_dump()
            )
            raise RuntimeError("LLM returned no choices")
        content = response.choices[0].message.content

        json_schema = None
        if llm_request.json_schema is not None:
            content_json = json_repair.loads(content, return_objects=True) 
            if len(content_json) == 0:
                raise json.JSONDecodeError("Invalid json", content, 0)
            if isinstance(content_json, list):
                json_schema = []
                for content in content_json:
                    json_schema.append(llm_request.json_schema(**content))
            elif llm_request.bulk:
                json_schema = [llm_request.json_schema(**content_json)]
            else:
                json_schema = llm_request.json_schema(**content_json)

I never expected the concepts from Fluent Python—especially data models, function factories, and decorators—to be this useful when designing an elegant system like this.

Seeing how these concepts can be combined to build a systematic and maintainable LLM infrastructure has been a good reminder that strong software engineering fundamentals remain highly relevant when building modern AI systems.

It also encourages me to keep improving my engineering skills by reading more deeply, understanding the fundamentals, and applying what I learn through real projects.

Building an Observable Agent Graph from Scratch

kai wen ng — Sun, 09 Aug 2026 07:36:30 +0000

Observability is important for LLM agents. Understanding the decision process, node execution, latency, and success rate of individual calls becomes increasingly important in a production system.

Instead of simply relying on existing tools such as Langfuse and LangGraph, I wanted to understand what happens underneath and build a customised solution for my own use case.

The Graph

The first version of the graph supports three fundamental execution mechanisms:

Node execution and directed edges
Conditional routing
Fan-out and fan-in execution

This design follows the principle described by Bloch that “All programmers are API designers. Good programs are modular, and intermodular boundaries define APIs. Good modules get reused.” The graph therefore treats each node as an independent computational unit with a well-defined interface to the rest of the graph.

For simplicity, each node is initially restricted to a single outgoing transition. Sequential execution can therefore be represented by returning a single node name, while more complex execution patterns are represented explicitly through routing or fan-out mechanisms.

Each node receives the current graph state and returns a StateUpdate containing the state fields that should be modified. The same principle is applied to parallel execution. A fan-out operation can execute multiple nodes against the same input state, after which their resulting state updates are collected and merged. The fan-in mechanism is responsible for combining these updates while preventing unintended state overwrites.

The graph structure can consequently be represented using a collection of nodes, edges, routers, fan-outs, and fan-ins:

@dataclass
class Edge:
    source: str
    target: str

@dataclass
class FanOut:
    source: str
    targets: list[str]

@dataclass
class FanIn:
    sources: list[str]
    target: str

@dataclass
class StateUpdate:
    values: dict[str, Any] = field(default_factory=dict)
    observation: dict[str, Any] = field(default_factory=dict)

@dataclass
class Node:
    name: str
    func: Callable[[Any], Awaitable[StateUpdate]]

After defining the data classes, we implemented the graph itself with a set of unified functions for adding nodes, edges, routers, fan-out, and fan-in operations. Each function performs type checking and basic validation when the corresponding graph component is added, ensuring that invalid graph structures are detected as early as possible.

class Graph:
    def __init__(self):
        self.nodes: dict[str, Node] = {}
        self.edges: dict[str, Edge] = {}
        self.routers: dict[str, Callable] = {}
        self.fan_out: dict[str, FanOut] = {}
        self.fan_in: dict[str, FanIn] = {}
        self.compiled: CompiledGraph | None = None

    def add_node(self, name, func):
        self.nodes[name] = Node(name, func)

    def add_edge(self, source, target):
        if source in self.edges:
            raise ValueError(f"Node '{source}' already has an edge")
        self.edges[source] = Edge(source, target)

    def add_conditional_edges(self, source, router):
        if source in self.routers:
            raise ValueError(f"Node '{source}' already has a router")
        self.routers[source] = router

    def add_fanout(self, source, targets):
        if source in self.fan_out:
            raise ValueError(f"Node '{source}' already has a fanout")
        self.fan_out[source] = FanOut(source, targets)

    def add_fanin(self, sources, target):
        if target in self.fan_in:
            raise ValueError(f"Target '{target}' already has a fanin")
        self.fan_in[target] = FanIn(sources, target)

Graph Construction

The graph is constructed by registering nodes and their corresponding transitions. Nodes and edges are maintained as dictionaries of their respective dataclass representations. Routers are represented separately as callables because their output depends on the current execution state.
Validation checks the structural correctness of the graph. For example, the graph must contain valid START and END nodes, and every edge must reference nodes that are explicitly defined within the graph. This ensures that both the source and destination nodes of every edge are valid and prevents the execution engine from attempting to traverse undefined nodes at runtime.


    def _validate(self):
        if "START" not in self.nodes:
            raise ValueError("START node is required")

        if "END" not in self.nodes:
            raise ValueError("END node is required")

        for source, edge in self.edges.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown source node: {source}")

            if edge.source != source:
                raise ValueError(
                    f"Edge source mismatch: {source} != {edge.source}"
                )

            if edge.target not in self.nodes:
                raise ValueError(f"Unknown target node: {edge.target}")

        for source, fanout in self.fan_out.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown fan-out source: {source}")

            if fanout.source != source:
                raise ValueError(
                    f"Fan-out source mismatch: {source} != {fanout.source}"
                )

            for target in fanout.targets:
                if target not in self.nodes:
                    raise ValueError(f"Unknown fan-out target: {target}")

Graph Execution

When the compiled graph is executed, the initial state is passed into the graph executor at runtime. The state therefore belongs to an individual graph execution rather than to the LLM agent itself. The graph executor is responsible for determining which graph element should execute next as pass through the _execute_node for execution. To unify the response for each node, every node is expected to return a StateUpdate, except for the terminal END node. An empty values dictionary represents a valid execution in which the node does not modify the graph state.

@dataclass
class CompiledGraph:
    nodes: dict[str, Node]
    edges: dict[str, str]
    routers: dict[str, Callable[[Any], str | list[str]]]
    fanouts: dict[str, list[str]]
    fanins: dict[str, str]
    repo: ObservationRepository
    start: str = "START"
    end: str = "END"
    trace_id: str = None

    async def run(self, state):
        return await self._execute(self.start, state)

    async def _execute_node(self, name, state):
        try:
            node = self.nodes[name]
            start = time.perf_counter()
            update = await node.func(state)
            elapsed = time.perf_counter() - start
            await self._record_observation(
                name=name,
                elapsed=elapsed,
                update=update,
            )
            return update

        except Exception as e:
            await self._record_observation(
                name=name,
                elapsed=time.perf_counter() - start,
                update=StateUpdate(),
                success=False,
                error=str(e),
            )
            raise

    async def _execute(self, name, state):
        if name == self.end:
            return state

        update = await self._execute_node(name, state)
        state = self._apply_update(state, update)

        if name in self.routers:
            targets = self.routers[name](state)

            if isinstance(targets, str):
                return await self._execute(targets, state)

            return await self._execute_fanout(name, targets, state)

        if name in self.fanouts:
            return await self._execute_fanout(name, self.fanouts[name], state)

        edge = self.edges.get(name)

        if edge is None:
            raise ValueError(f"Node '{name}' has no outgoing edge")

        return await self._execute(edge.target, state)

    def _apply_update(self, state, update: StateUpdate):
        for field, value in update.values.items():
            if field not in state.model_fields:
                raise ValueError(f"Unknown state field: '{field}'")
            setattr(state, field, value)

        return state

When a fan-out is encountered, the graph executor resolves the target nodes and executes them through the same execute_node function. The fan-out therefore does not require a second implementation of logging, exception handling, or observation logic.

    async def _execute_fanout(self, source, targets, state):
        results = await asyncio.gather(
            *[self._execute_node(target, state) for target in targets]
        )

        update = self._merge_updates(results)
        state = self._apply_update(state, update)

        fanin = self.fanins.get(source)
        if fanin is None:
            return state

        return await self._execute(fanin.target, state)

    def _merge_updates(self, updates: list[StateUpdate]):
        fields = {}
        observations = []

        for update in updates:
            for field, value in update.values.items():
                if field in fields:
                    raise ValueError(f"State conflict: '{field}'")
                fields[field] = value

            if update.observation is not None:
                observations.append(update.observation)

        return StateUpdate(
            values=fields,
            observation={"fanout": observations} if observations else None,
        )

As all node executions must pass through _execute_node, it becomes straightforward to consistently record what happens during each node execution. This provides a single observation point where we can capture information such as the node name, input state, execution time, returned state update, execution status, and any exceptions that occur. More importantly, this approach ensures that observability is applied uniformly across all nodes without requiring individual nodes to implement their own logging or monitoring logic.

    async def _record_observation(
        self,
        name: str,
        elapsed: float,
        update: StateUpdate | None = None,
        success: bool = True,
        error: str | None = None,
    ):
        def _jsonable(value):
            if hasattr(value, "model_dump"):
                return value.model_dump(mode="json")
            return value
        metadata = update.observation
        metadata = {
            key: _jsonable(value)
            for key, value in metadata.items()
        }

        kwargs = {
            "node": name,
            "duration_ms": elapsed * 1000,
            "success": success,
            "observations": metadata,
            "error": error,
        }

        if self.trace_id is not None:
            kwargs["trace_id"] = self.trace_id

        record = AgentObservation(**kwargs)

        await self.repo.create(record)

        if self.trace_id is None:
            self.trace_id = record.trace_id

This implementation gives me a clearer understanding of how an agent graph can be constructed from relatively simple execution primitives. Although the current implementation is intentionally minimal, the same principles can be extended to more complex agent workflows. Nodes provide the computational units, edges define the execution topology, routers provide conditional transitions, and fan-out/fan-in mechanisms provide parallel execution and state aggregation.

Linux Memory Region Data Structures

kai wen ng — Sun, 09 Aug 2026 04:30:00 +0000

The memory descriptor (mm_struct) maintains information about a process's address space (current memory it owns). One of its members is a linked list of Virtual Memory Areas (VMAs), represented by vm_area_struct.
The memory descriptor (mm_struct) contains a field named mmap, which points to the head of the linked list of Virtual Memory Areas (VMAs). Each VMA (vm_area_struct) contains a pointer vm_next to the next memory region, allowing the kernel to traverse all virtual memory regions owned by the process.
The number of VMAs currently owned by the process is stored in map_count. The Linux kernel also imposes a maximum number of VMAs (65,536 by default), although most processes use far fewer.
A linked list works well when a process owns only a small number of memory regions. However, applications that frequently allocate memory (for example, through malloc(), shared libraries, or memory-mapped files) may end up with hundreds or thousands of VMAs.
Searching for a particular memory region by traversing the linked list requires examining each node until the desired region is found, giving a time complexity of O(n).

To improve lookup performance, Linux also stores the VMAs in a red-black tree. Since the tree remains approximately balanced, searching for a memory region requires only O(log n) time.

Each VMA contains an embedded red-black tree node (vm_rb of type struct rb_node). The root of the tree is stored in the process's memory descriptor (mm_rb).

The tree is ordered by the starting virtual address (vm_start) of each memory region:

Memory regions with a smaller starting address are placed in the left subtree.
Memory regions with a larger starting address are placed in the right subtree.

A red-black tree maintains balance using the following properties:

Every node is coloured either red or black.
The root node is always black.
If a node is red, both of its children must be black. (Two consecutive red nodes are not allowed.)
Every path from a node to any descendant NIL (null) leaf contains the same number of black nodes. The NIL leaves are conceptual nodes and are considered black.

These rules ensure that the tree remains balanced, limiting its height to at most 2 log₂(n + 1). As a result, searching, inserting, and deleting VMAs all have O(log n) time complexity, making them significantly faster than traversing a linked list when a process owns many memory regions.

Maple Tree (Linux 6.1+)

Starting from Linux 6.1, the kernel replaced both the linked list (mmap) and the red-black tree (mm_rb) used to manage Virtual Memory Areas (VMAs) with a new data structure called the Maple Tree.
Previously, the memory descriptor (mm_struct) contained both:

struct vm_area_struct *mmap;  // Linked list of VMAs
struct rb_root mm_rb;         // Red-black tree of VMAs

The linked list was used to iterate through all memory regions, while the red-black tree provided fast lookups. Every time a VMA was inserted or removed, both structures had to be updated.
In modern kernels, these fields have been replaced by a single Maple Tree:

struct maple_tree mm_mt;

The Maple Tree performs both tasks:

Fast searching for a memory region.
Efficient traversal of all VMAs. As a result, the kernel only needs to maintain one data structure instead of two.

How the Maple Tree works

Unlike a red-black tree, where each node stores only a single memory region, a Maple Tree stores multiple memory regions within each node.
For example, a red-black tree might look like:

        40
      /    \
    20      60
   /  \    /  \
 10 30  50 70

where each node stores one key.
A Maple Tree is conceptually more like:

+--------------------------------------+
| 10 | 20 | 30 | 40 | 50 | 60 | 70 |
+--------------------------------------+

where a single node stores multiple keys and pointers to child nodes. This is similar to a B-tree, although the Maple Tree has its own specialised implementation for managing ranges of virtual addresses.
Searching, inserting, and deleting VMAs continue to have a time complexity of O(log n), but with lower constant overhead and better real-world performance than the previous implementation.

Agent Graph from Scratch: Discussion Needed

kai wen ng — Sun, 09 Aug 2026 04:30:00 +0000

Building AI agents from scratch allows developers to have full control and customize every workflow.
Recently, in one of my projects, I built an agent with the following workflow:

Workflow Description

The agent begins at the START node, where the planning() function analyzes the user's request and determines the execution path. If the request can be answered immediately, the workflow proceeds to IMMEDIATE_REPLY. If the query requires refinement, it is first processed by CONSTRUCT_NEW_QUERY before continuing. Otherwise, execution moves directly to PLAN_EXECUTION.

During PLAN_EXECUTION, the agent prepares the retrieval strategy, followed by PROJECT_QUERY, which retrieves the relevant project information. The collected information is then used by the END node to generate a response for the user.

After generating the response, the workflow checks whether additional information is required. If so, EMBEDDING_QUERY performs semantic retrieval to gather more context before returning to END to regenerate the response. This retrieval loop continues until no further information is needed or the retry limit is reached, after which the workflow terminates through IMMEDIATE_REPLY.

State Management in Agentic Graphs

One thing I've started noticing while building graph-based agents is how quickly the state grows.
The workflow itself is straightforward: each node performs one task, routing edges decide where to go next, and every node appends its output back into the shared state. However, after a few iterations, the agent accumulates planning decisions, retrieval results, generated queries, intermediate responses, retry counters, and many other pieces of information.
To keep the state manageable, I grouped related fields into smaller Pydantic models instead of placing every attribute directly into one large state.

from pydantic import BaseModel

# Individual state containers
class PlanningState(BaseModel):
    ...
    # Stores routing decisions made during planning

class QueryState(BaseModel):
    ...
    # Stores structured search parameters

class ResponseState(BaseModel):
    ...
    # Stores the generated response and follow-up actions

# Main agent state
class AgentState(BaseModel):
    user_input: str
    chat_history: str | None = None

    planning: PlanningState | None = None
    query: QueryState | None = None
    response: ResponseState | None = None

    retrieved_projects: list | None = None
    retrieved_documents: list | None = None

    retry_count: int = 0

This keeps the top-level state relatively clean while still giving every node a single input/output object.

The concern I have is scalability. As the workflow becomes more sophisticated, the AgentState will inevitably contain more nested states. While accessing data remains simple (state.planning.xxx, state.query.xxx, etc.), it feels like I'm simply moving the complexity one level deeper rather than reducing it.

Another design decision I made is that each node should do exactly one thing. A node performs its task and updates the state—it does not contain if/else branching. All decision making is handled by the graph's routing edges. In other words:

Nodes perform work.
Edges decide where execution goes next.

This makes each node easier to test and reason about, but it also means the routing logic becomes increasingly complex as the graph grows.

I'm curious how others approach this problem:

How do you prevent the shared state from becoming a "God object"?
Do you split state by capability, by workflow stage, or something else?
Do you keep branching entirely in the graph, or allow nodes to contain some business logic and conditional execution?
At what point does a single graph become too large, and when do you split it into subgraphs or sub-agents?

I'd be interested to hear how others balance graph simplicity, state management, and maintainability as agentic workflows become more complex.

DEV Community: kai wen ng

Testing Production AI Agents: A Practical Framework for Graph-Based Agent Systems

Table of Contents

1. Introduction

2. Why Testing AI Agents Is Different

3. The Anatomy of a Production AI Agent

Layer 1: Deterministic application logic

Layer 2: Tools

Layer 3: LLM reasoning

Layer 4: Agent behaviour

4. Why Traditional Unit Testing Becomes Difficult

5. Testing at Different Levels

5.1 Deterministic Unit Testing

5.2 Tool-Level Testing

5.3 Graph / Orchestration Testing

5.4 End-to-End Scenario Testing

6. Separating Deterministic and Probabilistic Components

7. Testing Individual Tools Without LLM Calls

8. Designing an End-to-End Agent Test Set

9. Testing Complex Business Requirements

10. Testing Graph Routing and State Transitions

11. Testing Tool Selection and Tool Arguments

12. Testing Retrieval and Validation Loops

13. Testing Failure and Recovery Behaviour

14. Testing Structured LLM Outputs

15. Measuring Agent Quality

Accuracy

Tool-call accuracy

Routing accuracy

Retrieval quality

Reliability

Efficiency

Latency

16. Regression Testing for Agents

17. Building an Agent Evaluation Dataset

18. Mocking vs Real Dependencies

19. Controlling LLM Variability During Testing

20. Common Testing Mistakes

Testing everything through the LLM

Testing only individual functions

Using only exact answer matching

Testing only happy paths

Ignoring tool-call traces

21. A Practical Testing Architecture

22. Lessons Learned from Production Agent Development

23. Conclusion

Learning OCR Process from Scratch – 004: Parsing OCR Detection Output

1. Filtering Low-Confidence Regions

2. Finding the Contours

3. Filtering Small and Low-Confidence Regions

4. Ordering the Box Points

5. Calculating the Detection Confidence

6. Expanding the Detection Box

7. From Detection to Recognition

What I Learned

Learning OCR Process from Scratch – 002: From Image to ONNX Tensor

1. Loading the Image

2. Resizing the Image

3. Converting and Normalising the Image

4. Understanding the ONNX Input

5. Converting HWC to CHW

6. Defining the Tensor Shape

7. Creating the ONNX Runtime Tensor

The Interesting Part

SQLAlchemy One-to-Many Relationships

1. Foreign Key

2. relationship()

back_populates

Important distinction

3. Lazy Loading

4. Eager Loading

Memory trade-off

5. Cascade

Overall...

8 Things I learnt from JavaScript First Time Developing Frontend

1. The DOM is the interface between JavaScript and HTML

Finding elements

2. JavaScript can dynamically generate HTML

3. Event handlers connect user actions to application logic

Passing the triggering element

`back_populates`

`Object.entries()` — loop over object key-value pairs

`for...of` — loop over values

8. `Promise.all()` runs asynchronous operations concurrently