DEV Community: kai wen ng

Building an Inference Engine from Scratch - Part 2: Basics of Attention Mechanism

kai wen ng — Wed, 12 Aug 2026 04:30:00 +0000

Before building an LLM inference engine, I need to understand the fine-grained details behind how modern LLMs work.

The book "Build a Large Language Model from Scratch" has been an excellent companion throughout this journey:

https://www.manning.com/books/build-a-large-language-model-from-scratch

It not only explains the theory behind Transformers, but also provides a practical understanding of how each component connects together to form a working language model.

Why Transformers?

Before Transformers, many sequence models were based on RNN architectures.
However, RNNs have several limitations:

Sequential computation makes training and inference slow.
Long sequences suffer from information loss due to limited hidden-state memory.
Parallelization is difficult.

What is Attention?

At its core, attention allows each token to determine which other tokens are important when building its representation.
Starting from token embeddings:

Token Embeddings
        |
        |
Linear Projection
        |
   --------------
   |     |      |
   Q     K      V

Each embedding is transformed into:

Query (Q): What information is this token looking for?
Key (K): What information does this token contain?
Value (V): What information should be passed forward?

For each token:

The query vector is compared with all key vectors using dot product:

Attention Score = Q × Kᵀ

The scores are normalized using softmax:

Attention Weight = Softmax(Attention Score)

The weights are applied to the value vectors:

Context Vector = Attention Weight × V

The result is a context-aware representation where each token contains information from other relevant tokens.

Multi-Head Attention

Instead of learning only one attention pattern, Transformers use multiple attention heads.
Each head can learn different relationships:

Syntax relationships
Long-distance dependencies
Semantic similarity
Position-based patterns

The outputs from all heads are combined to create a richer representation.

Causal Attention

For autoregressive language models, the model should only use previous tokens when predicting the next token.

Example:

The cat sat on the ...

When predicting the next word, the model should not have access to future tokens.
This is achieved using a causal mask:

Before Softmax:

[0    -inf  -inf]
[0     0    -inf]
[0     0     0 ]

The -inf values become zero probability after applying softmax.

Dropout

Dropout is a regularization technique used during training. During training, some activations are randomly disabled so the model does not depend too heavily on specific neurons or pathways. This encourages the network to learn more robust representations and reduces overfitting. During inference, dropout is disabled.

Layer Normalization

Layer normalization stabilizes activations throughout the network. It normalizes hidden representations and applies learnable:

Scale parameters
Shift parameters

This helps maintain stable signal propagation through deep Transformer layers.

Activation Function: GELU

Modern Transformers commonly use GELU instead of ReLU.
ReLU:

ReLU(x) = max(0, x)

completely removes negative values.
GELU provides a smoother activation function, allowing small negative inputs to contribute instead of being completely discarded which in turn improved representation learning in deep networks.

Hidden Dimension / Neurons

The hidden dimension determines the size of the representation space.
A larger hidden dimension allows the model to capture more complex patterns and relationships.

However:

More parameters require more memory.
More computation is required.
Larger models require more training data.

Residual Connections (Shortcut Connections)

Transformer blocks contain residual connections:

Output = Layer(x) + x

These shortcut paths:

Improve gradient flow.
Allow deeper networks to train effectively.
Preserve information between layers.

This concept was introduced in residual networks and became a fundamental component of modern deep learning architectures.

Transformer Block

A Transformer is not only an attention mechanism.
A typical Transformer block contains:

Input
 |
Layer Normalization
 |
Multi-Head Attention
 |
Residual Connection
 |
Layer Normalization
 |
Feed Forward Network
 |
Residual Connection
 |
Output

Keeping input and output dimensions consistent allows multiple Transformer blocks to be stacked together.

Model Parameters

To understand the size of an LLM, we need to know how many trainable parameters it contains.
During model implementation, the number of parameters can be calculated using numel, which counts the total number of elements inside a tensor.
For example, each weight matrix and bias vector contributes to the overall parameter count:

Total Parameters = numel(all model tensors)

Logits and Token Generation

The final layer outputs logits.
Logits are raw scores before converting them into probabilities.

Applying softmax:

Logits → Softmax → Probability Distribution

gives the probability of each possible next token.
The simplest decoding method:

argmax(probability)

selects the highest probability token.
To generate more diverse outputs, modern LLMs often use sampling strategies on the probability distribution

Top-k sampling
Top-p sampling

Autoregressive Generation Loop

LLMs generate text one token at a time.

The process:

Input tokens are passed through the Transformer.
The model predicts the next token.
The predicted token is appended to the input.
The process repeats. This simple loop is the foundation behind text generation in modern LLMs.

Building an inference engine from scratch is giving me a deeper understanding of what happens behind every API call.
The goal is not only to run models, but to understand every matrix multiplication, tensor transformation, and design decision that makes modern LLMs possible.

Speeding Up Vector Search with Quantization & Vector Indices

kai wen ng — Tue, 11 Aug 2026 04:30:00 +0000

Unlike traditional databases, vector databases do not search for exact matches. Instead, they compare the similarity between high-dimensional embedding vectors. A naive implementation would compare the query vector against every vector in the database, which quickly becomes computationally expensive as the dataset grows.

To improve search speed, vector databases use specialised indexing algorithms that significantly reduce the number of vector comparisons while maintaining high retrieval accuracy.

Flat Index

The Flat index is the simplest approach. It performs a brute-force search by calculating the similarity between the query vector and every vector stored in the database.

IVF (Inverted File Index)

IVF speeds up vector search by first clustering the dataset using algorithms such as K-Means.
Instead of searching the entire database, the query is first assigned to the nearest cluster, and only vectors within that cluster (or a few nearby clusters) are searched.
Conceptually, it behaves like an inverted index which tells the search: this cluster has this vector

Cluster A
 ├── Vector 1
 ├── Vector 5
 └── Vector 12

Cluster B
 ├── Vector 2
 ├── Vector 7
 └── Vector 20

LSH (Locality-Sensitive Hashing)

Locality-Sensitive Hashing maps similar vectors into the same hash buckets with high probability.
Unlike traditional hash functions that minimise collisions, LSH intentionally increases collisions for similar vectors. It does this by projecting vectors onto multiple random hyperplanes and generating hash codes based on the projection results.
During retrieval, only vectors in the same hash buckets are compared instead of the entire database.

HNSW (Hierarchical Navigable Small World)

HNSW is one of the most popular vector indexing algorithms today because it offers an excellent balance between speed and accuracy.
It organises vectors into a hierarchical graph.

The upper layers contain only a small number of nodes connected by long-distance links, similar to highways connecting major cities.
Lower layers contain many more nodes with shorter connections, similar to local roads.

A search begins from the top layer and gradually moves down through increasingly detailed layers until it reaches the nearest neighbours.

Important parameters include:

M – Maximum number of roads connected to each city. More roads create a denser network, making it easier to find efficient routes.
efConstruction – Determines how thoroughly the road network is planned during construction. Higher values spend more time finding the best roads between cities, resulting in a better network.
efSearch – Determines how many possible routes are explored during navigation. Higher values consider more roads before deciding on the destination, improving accuracy at the cost of slightly longer search time.

Quantization

Even with efficient indexes, storing and comparing high-dimensional floating-point vectors remains expensive.
Quantization reduces both memory usage and computation by compressing vectors before indexing.

Scalar Quantization (SQ)

Scalar Quantization compresses each floating-point value individually.
For example:

Original: 32-bit floating-point numbers
Compressed: 8-bit integers

Product Quantization (PQ)

Product Quantization compresses vectors more aggressively.
Instead of compressing each value independently, it:

Splits each embedding into multiple smaller sub-vectors.
Runs clustering on each sub-vector independently.
Stores only the centroid IDs instead of the original floating-point values.

During retrieval:

The query embedding is split into the same sub-vectors.
Distances are computed using the centroid codebooks instead of the original vectors.

Additional Optimisations

Vector search is rarely used alone in production systems. Several additional techniques are commonly combined to improve both performance and retrieval quality.

Metadata Filtering

Before performing vector search, metadata can eliminate documents that are impossible matches.
Examples include:

File type
Language
Date range
User permissions
Product category

Reducing the candidate set before vector search significantly improves efficiency.

Hybrid Search

Many production retrieval systems combine vector search with traditional keyword search.
Keyword search provides exact lexical matching, while vector search captures semantic similarity.
The results are then combined using ranking algorithms, often producing higher-quality retrieval than either approach alone.

Reference: https://www.oreilly.com/library/view/vector-databases/9781098177584/

How Maple Tree Manages VMAs

kai wen ng — Mon, 10 Aug 2026 04:30:00 +0000

Starting from Linux 6.1, the kernel replaced both the linked list (mmap) and the red-black tree (mm_rb) used to manage Virtual Memory Areas (VMAs) with a new data structure called the Maple Tree.
Maple Tree is a range-based tree structure designed for storing non-overlapping ranges.
It provides:

Fast lookup
Efficient traversal
Better memory usage
Read Copy and Update (RCU) safe operations Unlike a red-black tree, Maple Tree stores ranges directly, which matches how VMAs are represented.

Maple Tree Structure

Maple Tree nodes contain:

Slots
Pivots Each node 15x pivots and 16x entries. A slot stores an entry, while pivots define the range boundaries. The range information is implied by the pivot values instead of storing every range boundary explicitly. This allows Maple Tree to store large numbers of memory ranges efficiently.

Replace Operation

When modifying a Maple Tree node:

Maple Tree locates the node containing the target range.
A new empty node is allocated.
Existing entries and pivots before the modified range are copied into the new node.
The node is split at the position where the new range will be inserted.
The new entry is inserted into the correct slot.
The remaining entries and pivots after the modified range are copied into the new node.
The original node is replaced with the newly created node.

This replacement approach allows Maple Tree to support safe concurrent reads using RCU. Readers can continue accessing the old node while the writer prepares the updated version. Once the new node is complete, the kernel switches the reference to the new node.

Node Splitting

When a Maple Tree node becomes full and cannot store another entry:

The full node is removed from the active tree structure.
Two new nodes are created.
Existing entries and pivots are redistributed between the two new nodes.
The new entry is inserted into the correct position during the redistribution process.
The parent node is updated with the new child nodes and new pivot values. The split operation tries to keep the data density balanced between the two nodes while avoiding unnecessary tree height growth. After splitting, the parent node receives an additional slot pointing to the new node. The new pivot information is updated to represent the new range boundaries.

Maple Tree tries to maintain:

High Data Density: Nodes should contain many entries to reduce memory usage and improve cache efficiency.
Lower Tree Height: A shorter tree reduces the number of steps required during lookup.
Avoid Underfilled Nodes: After deletion or modification, entries may be redistributed or merged to avoid wasting space.

When multiple entries can be represented more efficiently as a single range:

A new node is created for the reorganized entries.
Existing entries are copied from the previous nodes.
Entries that can be represented as a continuous range are compressed into a single slot and pivot.
Remaining entries are moved together with neighbouring nodes.
The parent node is updated with the new node layout and pivot information.

This reduces the number of slots required and keeps the tree compact.
The goal of rebalancing is to maintain efficient storage by reducing unnecessary nodes while keeping lookup performance high.

Special thanks to the Linux Foundation and Liam Howlett's talk "The Maple Tree: Structure and Algorithms", which helped me understand the internal design and algorithms behind Maple Tree.
Link: https://www.youtube.com/watch?v=RaXhP-QLUxI

Towards the Stability of LLM Agents

kai wen ng — Sun, 09 Aug 2026 09:02:47 +0000

An industry-level LLM agent is not simply an API call that returns a response. It needs to be resilient to transient failures, malformed outputs, and schema violations.
To improve the stability of my agent system, I introduced two decorators around my LLM calls. They handle three common failure modes:

JSON decoding errors — provide the model with its previous malformed response and ask it to regenerate valid JSON.
Pydantic BaseModel validation errors — provide the model with the specific schema validation errors and ask it to correct the response.
429 rate-limit errors — retry with appropriate backoff.

This separates the reliability logic from the individual agent functions, allowing each LLM call to focus on its own task while the decorators provide a consistent recovery mechanism.

Json and Pydantic Basemodel

To handle problematic model outputs, I first use the json-repair package:
json-repair
PS: If you find the package useful, consider giving the author a ⭐ to show some support.

This allows the system to recover from common JSON formatting issues in the model's response. If the repaired result is still empty, I raise a JSONDecodeError manually.

This is useful because the retry decorator can then intercept the exception and provide the model with a meaningful retry message. Instead of blindly asking the model to try again, I can tell it:

Your previous response was not valid JSON.

and include the previous response through e.doc. This gives the model its failed output as context and allows it to regenerate the response correctly.

The next case is Pydantic ValidationError. A response can be valid JSON but still fail to match the expected schema. In this case, simply asking the model to retry is also not particularly useful. The retry message should contain the specific fields that failed validation and the expected constraints.

This information is available through e.errors(). I can extract the field location and validation message, then feed those details back to the model. This turns the retry from a blind regeneration into a targeted correction.


def llm_retry(max_retries=3):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            retry_message = None

            for attempt in range(max_retries):
                try:
                    return await func(*args, retry_message=retry_message, **kwargs)

                except ValidationError as e:
                    errors = [
                        {"field": ".".join(map(str, x["loc"])), "message": x["msg"]}
                        for x in e.errors()
                    ]
                    retry_message = (
                        "Your previous response failed schema validation.\n\n"
                        f"Validation errors:\n{errors}\n\n"
                        "Correct these errors and return the complete response as JSON only."
                    )

                except json.JSONDecodeError as e:
                    retry_message = (
                        "Your previous response was not valid JSON.\n\n"
                        f"Previous response:\n{e.doc}\n\n"
                        "Regenerate the complete response as valid JSON only. "
                        "Do not include Markdown or explanations."
                    )

            raise RuntimeError(f"LLM failed after {max_retries} attempts")

        return wrapper
    return decorator

The 429 Rate-Limit Decorator

This part is relatively straightforward. I use a simple function factory to create a decorator that wraps around the LLM calls and automatically retries requests that receive a 429 Too Many Requests response.

The decorator applies an exponential backoff between attempts, allowing the system to recover from temporary rate limiting without requiring each LLM call to implement its own retry logic.

def Retry429(trials=3, delay=1.0, backoff=2.0):
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            current_delay = delay

            for attempt in range(trials):
                try:
                    return await func(*args, **kwargs)
                except RateLimitError:
                    pass
                except httpx.HTTPStatusError as e:
                    if e.response.status_code != 429:
                        raise
                except httpx.HTTPError:
                    raise

                if attempt == trials - 1:
                    raise

                await asyncio.sleep(current_delay)
                current_delay *= backoff

        return wrapper

    return decorator

After having these decorators in place, I can define an LLM call as a simple function. Each function is responsible only for constructing the input arguments and returning the output arguments, represented by LLMRequest and LLMResponse.

Defining these interfaces as Pydantic BaseModels provides a consistent contract between different parts of the system and makes the overall agentic architecture more systematic and maintainable.

More importantly, the retry and recovery mechanisms are no longer coupled to individual LLM implementations. This allows me to support different types of LLM calls, such as image generation and VLM inference, without copy-pasting the same retry and validation logic across every function. The individual functions remain focused on their specific task, while the shared decorators handle the common reliability concerns.


    @Retry429(trials=10, delay=10)
    @llm_retry()
    async def generate(self, llm_request: LLMRequest, retry_message=None)-> LLMResponse:
        args = {}
        # Just some args parsing
        if retry_message is not None:
            args["messages"].append({"role": "system", "content": retry_message})

        response = await self.client.chat.completions.create(**args)
        self.add_token_usage(response)
        if not response:
            raise RuntimeError("LLM returned empty response")

        if not response.choices:
            logger.error(
                "LLM returned no choices. Response=%s",
                response.model_dump()
            )
            raise RuntimeError("LLM returned no choices")
        content = response.choices[0].message.content

        json_schema = None
        if llm_request.json_schema is not None:
            content_json = json_repair.loads(content, return_objects=True) 
            if len(content_json) == 0:
                raise json.JSONDecodeError("Invalid json", content, 0)
            if isinstance(content_json, list):
                json_schema = []
                for content in content_json:
                    json_schema.append(llm_request.json_schema(**content))
            elif llm_request.bulk:
                json_schema = [llm_request.json_schema(**content_json)]
            else:
                json_schema = llm_request.json_schema(**content_json)

I never expected the concepts from Fluent Python—especially data models, function factories, and decorators—to be this useful when designing an elegant system like this.

Seeing how these concepts can be combined to build a systematic and maintainable LLM infrastructure has been a good reminder that strong software engineering fundamentals remain highly relevant when building modern AI systems.

It also encourages me to keep improving my engineering skills by reading more deeply, understanding the fundamentals, and applying what I learn through real projects.

Building an Observable Agent Graph from Scratch

kai wen ng — Sun, 09 Aug 2026 07:36:30 +0000

Observability is important for LLM agents. Understanding the decision process, node execution, latency, and success rate of individual calls becomes increasingly important in a production system.

Instead of simply relying on existing tools such as Langfuse and LangGraph, I wanted to understand what happens underneath and build a customised solution for my own use case.

The Graph

The first version of the graph supports three fundamental execution mechanisms:

Node execution and directed edges
Conditional routing
Fan-out and fan-in execution

This design follows the principle described by Bloch that “All programmers are API designers. Good programs are modular, and intermodular boundaries define APIs. Good modules get reused.” The graph therefore treats each node as an independent computational unit with a well-defined interface to the rest of the graph.

For simplicity, each node is initially restricted to a single outgoing transition. Sequential execution can therefore be represented by returning a single node name, while more complex execution patterns are represented explicitly through routing or fan-out mechanisms.

Each node receives the current graph state and returns a StateUpdate containing the state fields that should be modified. The same principle is applied to parallel execution. A fan-out operation can execute multiple nodes against the same input state, after which their resulting state updates are collected and merged. The fan-in mechanism is responsible for combining these updates while preventing unintended state overwrites.

The graph structure can consequently be represented using a collection of nodes, edges, routers, fan-outs, and fan-ins:

@dataclass
class Edge:
    source: str
    target: str

@dataclass
class FanOut:
    source: str
    targets: list[str]

@dataclass
class FanIn:
    sources: list[str]
    target: str

@dataclass
class StateUpdate:
    values: dict[str, Any] = field(default_factory=dict)
    observation: dict[str, Any] = field(default_factory=dict)

@dataclass
class Node:
    name: str
    func: Callable[[Any], Awaitable[StateUpdate]]

After defining the data classes, we implemented the graph itself with a set of unified functions for adding nodes, edges, routers, fan-out, and fan-in operations. Each function performs type checking and basic validation when the corresponding graph component is added, ensuring that invalid graph structures are detected as early as possible.

class Graph:
    def __init__(self):
        self.nodes: dict[str, Node] = {}
        self.edges: dict[str, Edge] = {}
        self.routers: dict[str, Callable] = {}
        self.fan_out: dict[str, FanOut] = {}
        self.fan_in: dict[str, FanIn] = {}
        self.compiled: CompiledGraph | None = None

    def add_node(self, name, func):
        self.nodes[name] = Node(name, func)

    def add_edge(self, source, target):
        if source in self.edges:
            raise ValueError(f"Node '{source}' already has an edge")
        self.edges[source] = Edge(source, target)

    def add_conditional_edges(self, source, router):
        if source in self.routers:
            raise ValueError(f"Node '{source}' already has a router")
        self.routers[source] = router

    def add_fanout(self, source, targets):
        if source in self.fan_out:
            raise ValueError(f"Node '{source}' already has a fanout")
        self.fan_out[source] = FanOut(source, targets)

    def add_fanin(self, sources, target):
        if target in self.fan_in:
            raise ValueError(f"Target '{target}' already has a fanin")
        self.fan_in[target] = FanIn(sources, target)

Graph Construction

The graph is constructed by registering nodes and their corresponding transitions. Nodes and edges are maintained as dictionaries of their respective dataclass representations. Routers are represented separately as callables because their output depends on the current execution state.
Validation checks the structural correctness of the graph. For example, the graph must contain valid START and END nodes, and every edge must reference nodes that are explicitly defined within the graph. This ensures that both the source and destination nodes of every edge are valid and prevents the execution engine from attempting to traverse undefined nodes at runtime.


    def _validate(self):
        if "START" not in self.nodes:
            raise ValueError("START node is required")

        if "END" not in self.nodes:
            raise ValueError("END node is required")

        for source, edge in self.edges.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown source node: {source}")

            if edge.source != source:
                raise ValueError(
                    f"Edge source mismatch: {source} != {edge.source}"
                )

            if edge.target not in self.nodes:
                raise ValueError(f"Unknown target node: {edge.target}")

        for source, fanout in self.fan_out.items():
            if source not in self.nodes:
                raise ValueError(f"Unknown fan-out source: {source}")

            if fanout.source != source:
                raise ValueError(
                    f"Fan-out source mismatch: {source} != {fanout.source}"
                )

            for target in fanout.targets:
                if target not in self.nodes:
                    raise ValueError(f"Unknown fan-out target: {target}")

Graph Execution

When the compiled graph is executed, the initial state is passed into the graph executor at runtime. The state therefore belongs to an individual graph execution rather than to the LLM agent itself. The graph executor is responsible for determining which graph element should execute next as pass through the _execute_node for execution. To unify the response for each node, every node is expected to return a StateUpdate, except for the terminal END node. An empty values dictionary represents a valid execution in which the node does not modify the graph state.

@dataclass
class CompiledGraph:
    nodes: dict[str, Node]
    edges: dict[str, str]
    routers: dict[str, Callable[[Any], str | list[str]]]
    fanouts: dict[str, list[str]]
    fanins: dict[str, str]
    repo: ObservationRepository
    start: str = "START"
    end: str = "END"
    trace_id: str = None

    async def run(self, state):
        return await self._execute(self.start, state)

    async def _execute_node(self, name, state):
        try:
            node = self.nodes[name]
            start = time.perf_counter()
            update = await node.func(state)
            elapsed = time.perf_counter() - start
            await self._record_observation(
                name=name,
                elapsed=elapsed,
                update=update,
            )
            return update

        except Exception as e:
            await self._record_observation(
                name=name,
                elapsed=time.perf_counter() - start,
                update=StateUpdate(),
                success=False,
                error=str(e),
            )
            raise

    async def _execute(self, name, state):
        if name == self.end:
            return state

        update = await self._execute_node(name, state)
        state = self._apply_update(state, update)

        if name in self.routers:
            targets = self.routers[name](state)

            if isinstance(targets, str):
                return await self._execute(targets, state)

            return await self._execute_fanout(name, targets, state)

        if name in self.fanouts:
            return await self._execute_fanout(name, self.fanouts[name], state)

        edge = self.edges.get(name)

        if edge is None:
            raise ValueError(f"Node '{name}' has no outgoing edge")

        return await self._execute(edge.target, state)

    def _apply_update(self, state, update: StateUpdate):
        for field, value in update.values.items():
            if field not in state.model_fields:
                raise ValueError(f"Unknown state field: '{field}'")
            setattr(state, field, value)

        return state

When a fan-out is encountered, the graph executor resolves the target nodes and executes them through the same execute_node function. The fan-out therefore does not require a second implementation of logging, exception handling, or observation logic.

    async def _execute_fanout(self, source, targets, state):
        results = await asyncio.gather(
            *[self._execute_node(target, state) for target in targets]
        )

        update = self._merge_updates(results)
        state = self._apply_update(state, update)

        fanin = self.fanins.get(source)
        if fanin is None:
            return state

        return await self._execute(fanin.target, state)

    def _merge_updates(self, updates: list[StateUpdate]):
        fields = {}
        observations = []

        for update in updates:
            for field, value in update.values.items():
                if field in fields:
                    raise ValueError(f"State conflict: '{field}'")
                fields[field] = value

            if update.observation is not None:
                observations.append(update.observation)

        return StateUpdate(
            values=fields,
            observation={"fanout": observations} if observations else None,
        )

As all node executions must pass through _execute_node, it becomes straightforward to consistently record what happens during each node execution. This provides a single observation point where we can capture information such as the node name, input state, execution time, returned state update, execution status, and any exceptions that occur. More importantly, this approach ensures that observability is applied uniformly across all nodes without requiring individual nodes to implement their own logging or monitoring logic.

    async def _record_observation(
        self,
        name: str,
        elapsed: float,
        update: StateUpdate | None = None,
        success: bool = True,
        error: str | None = None,
    ):
        def _jsonable(value):
            if hasattr(value, "model_dump"):
                return value.model_dump(mode="json")
            return value
        metadata = update.observation
        metadata = {
            key: _jsonable(value)
            for key, value in metadata.items()
        }

        kwargs = {
            "node": name,
            "duration_ms": elapsed * 1000,
            "success": success,
            "observations": metadata,
            "error": error,
        }

        if self.trace_id is not None:
            kwargs["trace_id"] = self.trace_id

        record = AgentObservation(**kwargs)

        await self.repo.create(record)

        if self.trace_id is None:
            self.trace_id = record.trace_id

This implementation gives me a clearer understanding of how an agent graph can be constructed from relatively simple execution primitives. Although the current implementation is intentionally minimal, the same principles can be extended to more complex agent workflows. Nodes provide the computational units, edges define the execution topology, routers provide conditional transitions, and fan-out/fan-in mechanisms provide parallel execution and state aggregation.

Linux Memory Region Data Structures

kai wen ng — Sun, 09 Aug 2026 04:30:00 +0000

The memory descriptor (mm_struct) maintains information about a process's address space (current memory it owns). One of its members is a linked list of Virtual Memory Areas (VMAs), represented by vm_area_struct.
The memory descriptor (mm_struct) contains a field named mmap, which points to the head of the linked list of Virtual Memory Areas (VMAs). Each VMA (vm_area_struct) contains a pointer vm_next to the next memory region, allowing the kernel to traverse all virtual memory regions owned by the process.
The number of VMAs currently owned by the process is stored in map_count. The Linux kernel also imposes a maximum number of VMAs (65,536 by default), although most processes use far fewer.
A linked list works well when a process owns only a small number of memory regions. However, applications that frequently allocate memory (for example, through malloc(), shared libraries, or memory-mapped files) may end up with hundreds or thousands of VMAs.
Searching for a particular memory region by traversing the linked list requires examining each node until the desired region is found, giving a time complexity of O(n).

To improve lookup performance, Linux also stores the VMAs in a red-black tree. Since the tree remains approximately balanced, searching for a memory region requires only O(log n) time.

Each VMA contains an embedded red-black tree node (vm_rb of type struct rb_node). The root of the tree is stored in the process's memory descriptor (mm_rb).

The tree is ordered by the starting virtual address (vm_start) of each memory region:

Memory regions with a smaller starting address are placed in the left subtree.
Memory regions with a larger starting address are placed in the right subtree.

A red-black tree maintains balance using the following properties:

Every node is coloured either red or black.
The root node is always black.
If a node is red, both of its children must be black. (Two consecutive red nodes are not allowed.)
Every path from a node to any descendant NIL (null) leaf contains the same number of black nodes. The NIL leaves are conceptual nodes and are considered black.

These rules ensure that the tree remains balanced, limiting its height to at most 2 log₂(n + 1). As a result, searching, inserting, and deleting VMAs all have O(log n) time complexity, making them significantly faster than traversing a linked list when a process owns many memory regions.

Maple Tree (Linux 6.1+)

Starting from Linux 6.1, the kernel replaced both the linked list (mmap) and the red-black tree (mm_rb) used to manage Virtual Memory Areas (VMAs) with a new data structure called the Maple Tree.
Previously, the memory descriptor (mm_struct) contained both:

struct vm_area_struct *mmap;  // Linked list of VMAs
struct rb_root mm_rb;         // Red-black tree of VMAs

The linked list was used to iterate through all memory regions, while the red-black tree provided fast lookups. Every time a VMA was inserted or removed, both structures had to be updated.
In modern kernels, these fields have been replaced by a single Maple Tree:

struct maple_tree mm_mt;

The Maple Tree performs both tasks:

Fast searching for a memory region.
Efficient traversal of all VMAs. As a result, the kernel only needs to maintain one data structure instead of two.

How the Maple Tree works

Unlike a red-black tree, where each node stores only a single memory region, a Maple Tree stores multiple memory regions within each node.
For example, a red-black tree might look like:

        40
      /    \
    20      60
   /  \    /  \
 10 30  50 70

where each node stores one key.
A Maple Tree is conceptually more like:

+--------------------------------------+
| 10 | 20 | 30 | 40 | 50 | 60 | 70 |
+--------------------------------------+

where a single node stores multiple keys and pointers to child nodes. This is similar to a B-tree, although the Maple Tree has its own specialised implementation for managing ranges of virtual addresses.
Searching, inserting, and deleting VMAs continue to have a time complexity of O(log n), but with lower constant overhead and better real-world performance than the previous implementation.

Agent Graph from Scratch: Discussion Needed

kai wen ng — Sun, 09 Aug 2026 04:30:00 +0000

Building AI agents from scratch allows developers to have full control and customize every workflow.
Recently, in one of my projects, I built an agent with the following workflow:

Workflow Description

The agent begins at the START node, where the planning() function analyzes the user's request and determines the execution path. If the request can be answered immediately, the workflow proceeds to IMMEDIATE_REPLY. If the query requires refinement, it is first processed by CONSTRUCT_NEW_QUERY before continuing. Otherwise, execution moves directly to PLAN_EXECUTION.

During PLAN_EXECUTION, the agent prepares the retrieval strategy, followed by PROJECT_QUERY, which retrieves the relevant project information. The collected information is then used by the END node to generate a response for the user.

After generating the response, the workflow checks whether additional information is required. If so, EMBEDDING_QUERY performs semantic retrieval to gather more context before returning to END to regenerate the response. This retrieval loop continues until no further information is needed or the retry limit is reached, after which the workflow terminates through IMMEDIATE_REPLY.

State Management in Agentic Graphs

One thing I've started noticing while building graph-based agents is how quickly the state grows.
The workflow itself is straightforward: each node performs one task, routing edges decide where to go next, and every node appends its output back into the shared state. However, after a few iterations, the agent accumulates planning decisions, retrieval results, generated queries, intermediate responses, retry counters, and many other pieces of information.
To keep the state manageable, I grouped related fields into smaller Pydantic models instead of placing every attribute directly into one large state.

from pydantic import BaseModel

# Individual state containers
class PlanningState(BaseModel):
    ...
    # Stores routing decisions made during planning

class QueryState(BaseModel):
    ...
    # Stores structured search parameters

class ResponseState(BaseModel):
    ...
    # Stores the generated response and follow-up actions

# Main agent state
class AgentState(BaseModel):
    user_input: str
    chat_history: str | None = None

    planning: PlanningState | None = None
    query: QueryState | None = None
    response: ResponseState | None = None

    retrieved_projects: list | None = None
    retrieved_documents: list | None = None

    retry_count: int = 0

This keeps the top-level state relatively clean while still giving every node a single input/output object.

The concern I have is scalability. As the workflow becomes more sophisticated, the AgentState will inevitably contain more nested states. While accessing data remains simple (state.planning.xxx, state.query.xxx, etc.), it feels like I'm simply moving the complexity one level deeper rather than reducing it.

Another design decision I made is that each node should do exactly one thing. A node performs its task and updates the state—it does not contain if/else branching. All decision making is handled by the graph's routing edges. In other words:

Nodes perform work.
Edges decide where execution goes next.

This makes each node easier to test and reason about, but it also means the routing logic becomes increasingly complex as the graph grows.

I'm curious how others approach this problem:

How do you prevent the shared state from becoming a "God object"?
Do you split state by capability, by workflow stage, or something else?
Do you keep branching entirely in the graph, or allow nodes to contain some business logic and conditional execution?
At what point does a single graph become too large, and when do you split it into subgraphs or sub-agents?

I'd be interested to hear how others balance graph simplicity, state management, and maintainability as agentic workflows become more complex.

RAG Database with VLM as Extractor

kai wen ng — Fri, 07 Aug 2026 04:29:00 +0000

Problem

Building an effective semantic retrieval pipeline is challenging when the input data contains inconsistent formats, including:

Images
Random spreadsheets
PDFs and documents
Mixed structured and unstructured attachments

Traditional chunking methods struggle because the content structure varies significantly between files. Extracting meaningful text representations before embedding is required to improve retrieval quality.

Solution: VLM-Based Content Extraction

To handle heterogeneous input formats, the pipeline was redesigned to use a Vision Language Model (VLM) as an extraction layer.

Instead of relying on traditional parsers, each document page is converted into an image and processed independently by the VLM.

Pipeline Overview

Upload document attachment
Extract pages and convert each page into an image
Encode images into Base64 format
Send images to VLM for summarization and keyword extraction
Embed the generated summaries
Store embeddings in PostgreSQL with pgvector
Use semantic search for retrieval by the AI agent

Image Conversion Pipeline

Each page is treated as an independent chunk.

    async def vlm_inference(self, vlm_req: VLMRequest, file: UploadFile) -> list[str]:
        img_list = await extract_content(file, True)
        results = []

        for idx, img in enumerate(img_list):
            # Save image for inspection
            image_path = save_dir / f"page_{idx}.png"
            img.save(image_path)

            print(f"Saved image: {image_path}")

            # Convert to base64 for VLM
            img_base64 = image_to_base64(img)

            client = OllamaClient()
            result = await client.vlm_inference(
                vlm_req=vlm_req,
                image_base64=img_base64,
            )

            results.append(result)

        return results

VLM Client post:

    async def vlm_inference(
        self,
        vlm_req: VLMRequest,
        image_base64: str
    ) -> str:
        payload = {
            "model": vlm_req.model,
            "messages": [
                {
                    "role": "user",
                    "content": vlm_req.prompt,
                    "images": [image_base64],
                }
            ],
            "stream": False,
            "think": vlm_req.thinking,
        }
        response = requests.post(
            f"{LLM_BASE_URL}/api/chat",
            json=payload,
            timeout=300
        )

        response.raise_for_status()

        print(response.json())
        return response.json()['message']['content']

The advantage of page-level chunking is that it avoids complex document-specific parsing logic while preserving the visual context of the original file.

VLM Inference Client

The extracted page image is sent to the VLM through the Ollama API.

async def vlm_inference(
    self,
    vlm_req: VLMRequest,
    image_base64: str
) -> str:

    payload = {
        "model": vlm_req.model,
        "messages": [
            {
                "role": "user",
                "content": vlm_req.prompt,
                "images": [image_base64],
            }
        ],
        "stream": False,
        "think": vlm_req.thinking,
    }

    response = requests.post(
        f"{LLM_BASE_URL}/api/chat",
        json=payload,
        timeout=300
    )

    response.raise_for_status()

    return response.json()['message']['content']

Model Selection and Optimization

Initially, the extraction model used was:
Qwen3.5:20B
However, inference latency was too high for processing thousands of attachments.

The pipeline was optimized by switching to:

Qwen3.5:4B
The smaller model provided a better balance between:

Processing speed
Resource consumption
Extraction quality ## Thinking Mode Optimization

The Qwen model's reasoning mode introduced a practical issue.

With thinking enabled:

The model spent additional tokens on internal reasoning
Long generation consumed the output token budget
Some pages returned empty summaries because the model stopped after reaching the maximum output length

Increasing the token limit was not desirable because:

The system processes thousands of attachments
Larger outputs increase inference cost
Storage requirements increase significantly

Therefore, the reasoning mode was disabled.

With thinking disabled:

Response latency improved
Output became more concise
Important keywords were still preserved
Retrieval performance remained acceptable

Embedding and Storage

After VLM extraction, the generated summaries are converted into embeddings.

The embeddings are stored in PostgreSQL using pgvector.
The VLM-based extraction approach significantly improved semantic retrieval performance by creating consistent text representations from heterogeneous documents.

Result

The AI agent can now retrieve relevant information from:

Images
Scanned documents
Tables
Mixed-format attachments

without requiring individual parsers for every file type.

The key design decision was using the VLM as a universal extraction layer, transforming unstructured visual information into searchable semantic representations.

Building an LLM Inference Engine from Scratch: Tokenization Pipeline Notes

kai wen ng — Wed, 05 Aug 2026 04:30:00 +0000

The development of an LLM inference engine starts from understanding and implementing the tokenizer. Before the model can perform any computation, raw text input must be converted into numerical token IDs that correspond to entries in the model vocabulary.

Loading the Tokenizer Configuration

The first step was to load the tokenizer files downloaded from Hugging Face. The tokenizer configuration contains several important components:

Vocabulary (vocab.json): Maps token strings to integer token IDs.
Merge rules (merges.txt or equivalent JSON structure): Defines the Byte Pair Encoding (BPE) merge priority.
Regex pattern (tokenizer.json): Defines how raw text is initially split into smaller components.
Special tokens: Defines reserved tokens such as beginning-of-sequence, end-of-sequence, padding, etc.

Using the nlohmann::json library in C++, the vocabulary and merge dictionaries were parsed into native C++ data structures for efficient lookup.

Example structures:

std::unordered_map<std::string, int> vocabulary;

std::unordered_map<
    std::pair<std::string, std::string>,
    int,
    PairHash
> merge_rank;

The tokenizer class was designed to manage the complete encoding pipeline:

Raw Text
   |
   v
Normalization
   |
   v
Regex Pre-tokenization
   |
   v
Byte / Unicode Conversion
   |
   v
BPE Merge Algorithm
   |
   v
Vocabulary Lookup
   |
   v
Token IDs

Text Normalization and Pre-tokenization

Modern LLM tokenizers do not directly map words to vocabulary entries. Instead, they apply a multi-stage transformation.

The regex pattern stored in tokenizer.json is used to perform pre-tokenization.

For example, the pattern separates:

Words
Numbers
Punctuation
Whitespace
Contractions

A simplified example:
Input:

Hello world!

Regex output:

["Hello", " world", "!"]

Each segment is then processed independently by the BPE algorithm.

Byte-Level Encoding

Modern LLM tokenizers, such as GPT-2, Qwen, and many Hugging Face models, operate on bytes rather than directly on Unicode characters.

The original text:

你好

is first converted into UTF-8 bytes:

E4 BD A0
E5 A5 BD

Each byte is mapped into a special Unicode representation through a byte-to-unicode mapping.

The purpose of this mapping is to allow every possible byte value (0-255) to be represented as a valid Unicode token candidate.

Example:

Byte:
0xF0

Mapped Unicode:
ð

This creates an intermediate representation used by BPE.

Byte Pair Encoding (BPE) Merge Algorithm

The key discovery during implementation was that tokenization is not simply a vocabulary lookup.
The tokenizer does not immediately search:

"hello"

inside the vocabulary.
Instead, it performs iterative merging based on the merge rules.
Each pre-tokenized segment is first broken into individual byte/unicode units:
Example:

hello

becomes:

h e l l o

The tokenizer then checks adjacent pairs:

(h,e)
(e,l)
(l,l)
(l,o)

Each pair is searched in the merge dictionary.
The merge dictionary contains the priority ranking:

("h","e") -> 10
("he","l") -> 5
("hel","l") -> 3

A lower rank means a higher merge priority.

The algorithm repeatedly:

Finds all possible adjacent pairs.
Checks whether each pair exists in the merge dictionary.
Selects the pair with the lowest merge rank.
Combines the pair into a single token.
Repeats until no valid merges remain.

Vocabulary Lookup

After BPE merging is complete, the resulting token strings are searched in the vocabulary dictionary.

Example:
After merging:

["hello", "Ġworld"]

Vocabulary lookup:

hello     -> 15339
Ġworld    -> 1917

The final tokenizer output becomes:

These integer IDs are then used as input embeddings for the transformer model.

Repo: https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization

Loading Safetensors in C++

kai wen ng — Mon, 03 Aug 2026 04:30:00 +0000

First, use open() to open the file and check the file size.

fd = open(path.c_str(), O_RDONLY);
file_size = lseek(fd, 0, SEEK_END);

To avoid loading the entire file into memory, I use mmap() to map the file into virtual memory.

mapped = mmap(nullptr, file_size, PROT_READ, MAP_PRIVATE, fd, 0);

The first uint64_t in a safetensors file stores the header size.

uint64_t header_size;
memcpy(&header_size, mapped, sizeof(uint64_t));

char *header_ptr = static_cast<char *>(mapped) + sizeof(uint64_t);

std::string header(header_ptr, header_size);

The header contains the metadata information of each tensor. To parse the metadata, I use nlohmann::json.

metadata = nlohmann::json::parse(header);

The actual model parameters start after the header:

data_offset = sizeof(uint64_t) + header_size;

To access each layer's weights easily, I created a Tensor structure:

struct Tensor
{
    std::string name;
    std::string dtype;

    std::vector<int64_t> shape;

    uint64_t start;
    uint64_t end;

    char *data;
    size_t size;
};

A get_tensor() function is used to retrieve each tensor from the safetensors file. The important part is calculating the correct memory position using the tensor's data offsets.

Tensor get_tensor(const std::string &name)
{
    if (!metadata.contains(name))
        throw std::runtime_error("Tensor not found: " + name);

    auto json = metadata[name];

    Tensor tensor;

    tensor.name = name;
    tensor.dtype = json["dtype"];
    tensor.shape = json["shape"].get<std::vector<int64_t>>();

    tensor.start = json["data_offsets"][0].get<uint64_t>();
    tensor.end = json["data_offsets"][1].get<uint64_t>();

    tensor.size = tensor.end - tensor.start;

    tensor.data =
        static_cast<char *>(mapped) + data_offset + tensor.start;

    return tensor;
}

Since my PC has limited memory, I avoid loading the entire model weights. Instead, the weights remain inside the memory-mapped file, and the values are converted only when needed during inference. Note that this does not copy the actual parameters into memory; it only creates a pointer to the corresponding location inside the memory-mapped file.
During inference, the tensor data can be accessed by casting the raw memory pointer into the correct data type.

For BF16 weights:

embedding_weights = reinterpret_cast<uint16_t *>(tensor.data);
output.resize(embedding_dim);
size_t offset = token_id * embedding_dim;
for (int i = 0; i < embedding_dim; i++)
{
    output[i] = bf16_to_float(embedding_weights[offset + i]);
}

Alternatively, for faster inference, the tensor can be copied into allocated memory instead of accessing the memory-mapped file directly.

tensor.data = static_cast<char *>(malloc(tensor.size));
memcpy(
    tensor.data,
    static_cast<char *>(mapped) + data_offset + start,
    tensor.size
);

uint16_t *weights = reinterpret_cast<uint16_t *>(tensor.data);
for (int i = 0; i < size; i++)
{
    float value = bf16_to_float(weights[i]);
}

This loads the tensor data into normal allocated memory, allowing direct access during inference.

Impact of Inference Backends on LLM Reproducibility: Notes from a Research Paper

kai wen ng — Fri, 31 Jul 2026 04:30:00 +0000

Recently I read aboyt this article:
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

Here is what I learnt:
LLM inference engines provide a platform for running large language models on user hardware. However, the choice of inference engine can significantly affect model behaviour, introducing variations in output quality, reasoning length, and overall performance.

Beyond the model itself, the inference stack introduces additional factors that can influence results:

Default engine behaviour

Some inference engines, such as Ollama, automatically append default tokens, templates, or system prompts before the user-provided prompt.
Default generation parameters, such as repetition penalties or frequency penalties, may alter model behaviour. For example, default penalties in Ollama have been observed to reduce the length and depth of chain-of-thought reasoning in models such as DeepSeek.

Numerical and hardware-related variations

Prefix caching mechanisms are enabled by default in inference engines such as vLLM, Ollama, and SGLang. Although they are designed to reuse previously computed KV cache for improved efficiency, they can alter execution paths and the order of floating-point operations, leading to minor numerical variations.
Kernel-level tie-breaking during greedy decoding. When multiple tokens have identical logits (particularly under FP16 precision), PyTorch deterministically selects the lowest token ID, whereas LMDeploy's parallel Top-K kernel may select different tokens due to thread scheduling, resulting in divergent generation.
Accumulation precision during matrix multiplication. Inference engines differ in their choice of accumulation precision. For example, llama.cpp and Ollama may accumulate intermediate matrix multiplication results in FP32, whereas other engines perform accumulation in FP16. These differences introduce small rounding variations that can propagate through subsequent transformer layers and ultimately affect the generated output.

These factors are fundamentally manifestations of the same underlying issue: the inference stack is not merely an execution layer but part of the model's effective runtime environment. Small implementation differences can lead to measurable variations in generated outputs, especially for large reasoning models.

Therefore, simply fixing the random seed is insufficient to guarantee reproducibility. Deterministic behaviour requires controlling and documenting the complete inference stack, including:

Inference engine and version
Model format and quantisation method
Runtime backend and kernel implementations
Hardware configuration
Precision settings
Generation parameters
Prompt templates and system-level modifications

The inference environment should be treated as part of the model specification, and the complete system footprint should be exposed alongside evaluation results to ensure reproducibility and fair comparison between deployments.

Credit goes to the original authors. Their paper provides more detailed explanations, methodology, and comprehensive results that inspired this writeup.

Stop Using chmod 777: Understanding Linux File Permissions

kai wen ng — Mon, 27 Jul 2026 04:30:00 +0000

Are you one of those people who runs:

chmod 777 -R /folder

every time you encounter a Linux permission issue?
I am. 😂

It works. The error disappears. Problem solved... right?

Not exactly.

Giving everyone full permission is usually a sign that we are fixing the symptom instead of understanding the actual problem.

Let's take a look at how Linux file permissions really work.

The Three Permission Types

Every file and directory in Linux has three basic permissions:

Permission	Symbol	Value	Meaning
Read	`r`	4	View contents
Write	`w`	2	Modify contents
Execute	`x`	1	Execute/access

The numeric values are used when setting permissions with chmod.

For example:

rwx = 4 + 2 + 1 = 7
rw- = 4 + 2 + 0 = 6
r-x = 4 + 0 + 1 = 5

The Three User Categories

Linux permissions are divided into three groups:

u → user (file owner)
g → group (users belonging to the file group)
o → other (everyone else)

When you run:

ls -l

you might see:

-rwxrw-r-x

Let's break it down.

- rwx rw- r-x
│ │   │   │
│ │   │   └── Other users
│ │   └────── Group
│ └────────── Owner
└──────────── File type

The first character represents the file type:

-  regular file
d  directory

Understanding Numeric Permissions

Example:

-rwxrw-r-x

Owner:

rwx = 4 + 2 + 1 = 7

Group:

rw- = 4 + 2 + 0 = 6

Other:

r-x = 4 + 0 + 1 = 5

Therefore:

chmod 765 file

means:

Owner   → rwx
Group   → rw-
Other   → r-x

Changing Permissions with chmod

There are two common ways to use chmod.

Numeric mode

Example:

chmod 755 script.sh

Meaning:

Owner → rwx (7)
Group → r-x (5)
Other → r-x (5)

Result:

-rwxr-xr-x

2. Symbolic mode

Instead of calculating numbers, we can directly modify permissions.
Remove read permission from others:

chmod o-r file

Before:

-rwxr--r--

After:

-rwxr-----

Add execute permission for the owner:

chmod u+x file

Before:

-rw-r--r--

After:

-rwxr--r--

If no target is specified:

chmod +x file

it applies to all permission groups.
Before:

-rw-r--r--

After:

-rwxr-xr-x

Directory Permissions Are Different

One important detail:
Directories do not behave exactly like files.

For directories:

Permission	Meaning
`r`	List files inside
`w`	Create/delete files
`x`	Enter/access directory

For example:

chmod 755 folder

allows:

Owner: full access
Others: enter and read files

Without the execute permission:

chmod 644 folder

you may see the directory name, but you cannot access files inside.

Why `chmod 777` Is Usually a Bad Idea

chmod 777 gives:

Owner  → rwx
Group  → rwx
Other  → rwx

Meaning every user and process on the system can modify the files.

For example:

chmod 777 uploads/

might temporarily fix an application issue, but it can introduce security problems.

A better approach is usually to fix ownership:

chown user:group uploads/
chmod 755 uploads/

The correct permission depends on who actually needs access.

Next Time You See "Permission Denied"

Before running:

chmod 777 -R /folder

try checking:

ls -l

Then ask:

Who owns this file?
Which group owns it?
Does the user have the required permission?
Is the directory missing the execute permission?

Linux permissions are not complicated.

Once you understand:

three permission bits (rwx)
three ownership groups (ugo)
numeric permissions (755, 644, etc.)

most permission issues become much easier to debug.

Use the permission you need, not the maximum permission available.

DEV Community: kai wen ng

Building an Inference Engine from Scratch - Part 2: Basics of Attention Mechanism

Why Transformers?

What is Attention?

Multi-Head Attention

Causal Attention

Dropout

Layer Normalization

Activation Function: GELU

Hidden Dimension / Neurons

Residual Connections (Shortcut Connections)

Transformer Block

Model Parameters

Logits and Token Generation

Autoregressive Generation Loop

Speeding Up Vector Search with Quantization & Vector Indices

Flat Index

IVF (Inverted File Index)

LSH (Locality-Sensitive Hashing)

HNSW (Hierarchical Navigable Small World)

Quantization

Scalar Quantization (SQ)

Product Quantization (PQ)

Additional Optimisations

Metadata Filtering

Hybrid Search

How Maple Tree Manages VMAs

Maple Tree Structure

Replace Operation

Node Splitting

Towards the Stability of LLM Agents

Json and Pydantic Basemodel

The 429 Rate-Limit Decorator

Building an Observable Agent Graph from Scratch

The Graph

Graph Construction

Graph Execution

Linux Memory Region Data Structures

Maple Tree (Linux 6.1+)

How the Maple Tree works

Agent Graph from Scratch: Discussion Needed

Workflow Description

State Management in Agentic Graphs

RAG Database with VLM as Extractor

Problem

Solution: VLM-Based Content Extraction

Pipeline Overview

Image Conversion Pipeline

VLM Inference Client

Model Selection and Optimization

Embedding and Storage

Result

Building an LLM Inference Engine from Scratch: Tokenization Pipeline Notes

Loading the Tokenizer Configuration

Text Normalization and Pre-tokenization

Byte-Level Encoding

Byte Pair Encoding (BPE) Merge Algorithm

Vocabulary Lookup

Loading Safetensors in C++

Impact of Inference Backends on LLM Reproducibility: Notes from a Research Paper

Default engine behaviour

Numerical and hardware-related variations

Stop Using chmod 777: Understanding Linux File Permissions

The Three Permission Types

The Three User Categories

Understanding Numeric Permissions

Changing Permissions with chmod

Numeric mode

2. Symbolic mode

Directory Permissions Are Different

you may see the directory name, but you cannot access files inside.

Why chmod 777 Is Usually a Bad Idea

Next Time You See "Permission Denied"

Why `chmod 777` Is Usually a Bad Idea