DEV Community: ifnodoraemon

Building AI Agent Applications from Scratch

ifnodoraemon — Sun, 20 Sep 2026 06:40:41 +0000

What is an AI Agent?

An AI Agent is an AI system capable of perceiving its environment, making autonomous decisions, and executing actions. Unlike traditional single API calls, an Agent can:

🔄 Loop execution: Continuously adjust actions based on feedback
🛠️ Tool calling: Search the web, execute code, operate databases
📋 Task planning: Break down complex goals into actionable steps
🧠 Memory management: Maintain context and state over long conversations

Architecture Design

A complete AI Agent system typically includes the following core components:

graph LR
    subgraph Core["Core Layer"]
        LLM["Large Language Model"] --- PM["Prompt Manager"]
    end
    subgraph Decision["Decision Layer"]
        Router["Decision Engine"] --- Memory["Memory System"]
    end
    subgraph Tools["Tool Layer"]
        T1["Search"] & T2["Code Execution"] & T3["Database"] & T4["API"]
    end
    Core --> Decision --> Tools

Implementation Steps

Step 1: Install Dependencies

pip install langchain langchain-anthropic langgraph tavily-python

Step 2: Define Tools

from langchain.tools import tool
from datetime import datetime

@tool
def get_current_time() -> str:
    """Get the current time"""
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")

@tool
def search_web(query: str) -> str:
    """Search the web to get real-time information"""
    from tavily import TavilyClient
    client = TavilyClient()
    results = client.search(query, max_results=3)
    return "\n".join([r["content"] for r in results["results"]])

@tool
def execute_python(code: str) -> str:
    """Execute Python code and return the result"""
    import subprocess
    result = subprocess.run(
        ["python", "-c", code],
        capture_output=True, text=True, timeout=10
    )
    return result.stdout or result.stderr

Step 3: Building a State Machine with LangGraph (StateGraph)

Production-grade Agents in 2026 have completely abandoned black-box AgentExecutor abstractions. Instead, the paradigm has shifted to directed graphs centered around State Machines. This architecture provides extreme controllability and enforces strict data flow (State Schema).

from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode
from langchain_core.messages import AnyMessage, add_messages
from langchain_anthropic import ChatAnthropic

# 1. Strictly define the Agent's global state (State Schema)
class AgentState(TypedDict):
    messages: Annotated[list[AnyMessage], add_messages]
    current_task: str

# 2. Node Logic: The Model Reasoning Node
def call_model(state: AgentState):
    llm = ChatAnthropic(model="claude-sonnet-4-6-20260217")
    tools = [get_current_time, search_web, execute_python]
    llm_with_tools = llm.bind_tools(tools)
    response = llm_with_tools.invoke(state["messages"])
    return {"messages": [response]}

# 3. Build the Directed Graph
workflow = StateGraph(AgentState)
workflow.add_node("agent", call_model)
workflow.add_node("tools", ToolNode(tools)) # Built-in tool execution node

# 4. Define Routing and Edges
workflow.set_entry_point("agent")
# If the model's response includes tool_calls, route to tools; otherwise, END
workflow.add_conditional_edges("agent", lambda x: "tools" if x["messages"][-1].tool_calls else END)
workflow.add_edge("tools", "agent")

# 5. Compile into an Executable Application
app = workflow.compile()

Step 4: Execution and Tracing

With the graph architecture, we can precisely trace every state transition step:

inputs = {"messages": [("user", "Help me find the latest release date for foundation models.")]}
for event in app.stream(inputs, stream_mode="values"):
    message = event["messages"][-1]
    message.pretty_print() # Clearly inspect every step of Thought -> Action -> Observation

Model Selection Recommendations

Requirements	Recommended Model	Reason
Complex Agent tasks	Claude Opus 4.6	The strongest Agentic capabilities and Computer Use
Daily Agent development	Claude Sonnet 4.6	The best balance between speed and intelligence
Agents requiring deep reasoning	GPT-5.4 Thinking	Transparent thought chains, easier to debug
High-volume production environments	Gemini 3.1 Flash-Lite	Best cost-effectiveness

Best Practices

Make tool descriptions precise: LLMs decide when to call tools via their docstrings, the clearer the better.
Limit the number of tools: Keep it under 10 tools per Agent; too many will decrease selection accuracy. For large-scale external integrations, use the MCP Protocol for decoupling and standardized tool serving.
Add safety guardrails: Set explicit permissions and controls for sensitive tools like code execution or database operations.
Implement graceful degradation: When tool calls fail, the Agent should be able to identify the failure and switch strategies.
Monitoring and logging: Record the input and output of every tool call to facilitate debugging and optimization.

Enterprise-Grade Agent Architecture (Phase 2 Deep Dive)

In real-world production environments, an Agent might encounter API limits, database locks, or asynchronous jobs that take hours to complete. A naive synchronous architecture will instantly collapse (see our 7 Runtime Practices for Building AI Agents for in-depth operational patterns). Here are the most hardcore industry practices:

1. Cross-Session Persistence & Interruption Recovery (Redis Checkpointer)

To ensure an Agent remembers a user's context even after server restarts, or to pause execution pending human approval (Human-in-the-loop) before high-risk operations (like transferring funds), a Checkpointer is strictly mandatory. The high-concurrency standard for 2026 is using RedisSaver, achieving sub-millisecond state serialization.

from langgraph.checkpoint.redis import RedisSaver
import redis

# Establish the underlying Redis connection pool
redis_conn = redis.Redis(host='localhost', port=6379, db=0)

# Compile the Graph with Persistence
with RedisSaver(redis_conn) as checkpointer:
    app = workflow.compile(
        checkpointer=checkpointer,
        interrupt_before=["tools"] # Implement hard interruption before calling tools to wait for approval
    )

    # Use thread_id to differentiate user sessions, ensuring absolute concurrency state isolation
    config = {"configurable": {"thread_id": "user_10086_task_v2"}}

    # The Agent will automatically suspend before the 'tools' node; state is safely stored in Redis
    for event in app.stream(inputs, config=config):
        print(event)

2. Async Event-Driven Orchestration for Long-Running Tasks (Temporal)

Because standard HTTP requests usually timeout after 60 seconds, a synchronous blocking architecture will inevitably crash if your Agent needs to run in the background to scrape 100 webpages and generate a comprehensive financial report. The industry (e.g., OpenAI's Codex architecture) has shifted heavily towards using Temporal as the underlying durable workflow orchestration engine.

High-Concurrency Disaster Recovery Architecture:

API Gateway Dispatcher: Receives the user request and does not block execution; instead, it immediately returns a UUID ticket (Job_ID).
Temporal Worker Queue Listener: A fleet of background workers listens to the queue, spawning a dedicated LangGraph thread upon receiving the job.
Sleep and Automatic Polling: When the Agent must wait for an external webpage script to render, it calls await asyncio.sleep(300). Temporal physically serializes the entire in-memory state of the Agent into disk storage as a Checkpoint and releases CPU wait resources (100% Crash-safe).
Bidirectional State Webhooks: Once the task finishes, the final answer is precisely pushed back to the frontend browser via Webhooks or WebSockets.

3. Sandboxed Physical Isolation for Multi-modal & Multi-Agents

For a Coder Agent with code execution capabilities (like the execute_python tool from Step 2), it is utterly irresponsible to run its generated code directly on the host machine—this is highly susceptible to Prompt Injection attacks leading to catastrophic rm -rf scenarios. The only enterprise-grade solution is routing execution via gRPC to completely physically isolated, lightweight MicroVMs (like AWS Firecracker VMs or heavily restricted Docker sidecars) for sandbox execution. Even if an Agent "jailbreaks" and generates malicious commands, it will merely destroy a disposable sandbox with a nanosecond lifecycle, guaranteeing the absolute safety of the host application.

Conclusion

In 2026, building excellent AI Agents has long surpassed the infantile phase of "writing two lines of prompt and blindly calling an API." To actually deploy large language models to enterprise production lines and withstand millions of malicious requests and traffic spikes, it has evolved into a hyper-dimensional backend discipline merging exact state-machine topological design, centralized distributed scheduling, microsecond-level cache control, and OS-level sandbox defense.

Frequently Asked Questions (FAQ)

Q1: How can we extend an Agent's capabilities through external protocols?

In practical applications, Agents need to interact with various external systems. We recommend using the MCP Protocol, which completely decouples tool calling from business logic and provides a standardized extension mechanism. This improves security and simplifies architecture design.

Q2: How do Agents handle large-scale concurrent requests?

By combining Redis persistence mechanisms with distributed task orchestration engines (such as Temporal), Agents can achieve extremely strong concurrent processing and disaster recovery capabilities. This is much more stable than traditional monolithic synchronous architectures.

Originally published at Nobita Talks AI on blog.llmgo.top.

Agentic Loops & State Machine Design: From ReAct to Deterministic Control Flows

ifnodoraemon — Sun, 20 Sep 2026 06:40:30 +0000

When software engineers first experiment with Large Language Model (LLM) agents, the textbook pattern feels enchantingly simple: write a system prompt defining available tools, provide the LLM with a while (has_tool_call) loop, execute whatever functions the model outputs, and feed the results back into the conversation history.

In small-scale demos or isolated queries, this straightforward ReAct (Reasoning + Acting) loop performs adequately. However, when deployed into serious enterprise production—navigating dozens of heterogeneous tools, flaky external APIs, ill-formed data payloads, and complex multi-branch business logic—an unconstrained ReAct loop almost inevitably collapses.

The fundamental architectural flaw lies in a profound category mistake: Large Language Models are probabilistic next-token reasoners, not deterministic state controllers. Delegating execution control flow, state transitions, and retry loops entirely to probabilistic sampling is the production equivalent of running an unmonitored infinite loop on live traffic.

As the flagship opener of 《The Production AI Agent Architect Handbook》, this guide deconstructs why raw ReAct loops break at scale and demonstrates how to inject industrial-grade determinism, predictability, and resilience using Finite State Machines (FSM) and Graph-based Control Flows.

1. The Four Fatal Flaws of Raw ReAct Loops in Production

The classic ReAct framework interleaves model reasoning across three alternating phases: Thought $\rightarrow$ Action $\rightarrow$ Observation. The ubiquitous pseudo-code skeleton looks like this:

messages = [SystemMessage, UserPrompt]

while not is_done:
    response = llm.chat(messages, tools=tools)
    if response.has_tool_calls():
        for call in response.tool_calls:
            result = execute_tool(call.name, call.args)
            messages.append(ToolMessage(call_id=call.id, content=result))
    else:
        is_done = True
        return response.content

While elegant in an academic sandbox, this naive loop triggers four catastrophic failure modes under production concurrency:

1. Infinite Tool Retry Storms

When a downstream service throws an error—such as an HTTP 400 Bad Request due to a schema mismatch—the LLM typically reflects: "The arguments might be slightly malformed, let me try again." Without deterministic circuit breakers, probabilistic sampling causes the model to regenerate essentially identical requests with superficial variations. Within 30 seconds, an agent can churn through 20 tool calls, burning through hundreds of dollars in API tokens before being forcibly terminated by an HTTP gateway timeout.

2. Context Snowball & Attention Dilution

Every invocation argument and raw tool response (often sprawling JSON or scraped HTML blobs) is appended indiscriminately to the messages array. As multi-step tasks reach 15 or 20 iterations, prompt volume easily surges beyond 100,000 tokens.
As documented in our guide to Context Engineering in Practice, this causes extreme latency spikes and triggers the "Lost in the Middle" phenomenon. The LLM loses its attentional grip on the user's primary objectives, hallucinating random tool calls completely decoupled from the original goal.

3. Cascading Hallucinations & Dirty State Contamination

Without typed contract boundaries between steps, soft tool errors (e.g., an SQL query returning an empty set [] or unstructured error strings) are ingested directly into the reasoning loop. The model hallucinates plausible-looking placeholder records and feeds that corrupted state into downstream mutating operations—such as executing financial transactions or modifying customer accounts.

4. Lack of Deterministic Rollback & Escape Hatches

Traditional backend services rely on atomic transaction rollbacks and well-defined fallback handlers when unexpected states occur. In an unconstrained ReAct loop, because no formal transition graph exists, the agent cannot rewind to the last known safe checkpoint, nor can it cleanly hand off control to a human expert. It remains trapped in an erroneous trajectory until budget exhaustion.

2. From Freeform Loops to Graph Control: FSM Architecture

The remedy to these failure modes is strict Separation of Concerns:

The Large Language Model is the "Actor" (handling localized reasoning and unstructured extraction within a node), while the Finite State Machine is the "Director" (governing topological transitions, guards, and safety contracts).

An LLM must never act as both actor and director. The macroscopic routing graph must be defined statically in deterministic code, restricting model agency to strictly bounded state envelopes.

1. Production State Decomposition

A robust enterprise agent decomposes execution into discrete, verifiable lifecycle states:

stateDiagram-v2
    [*] --> Idle: User Request Ingested
    Idle --> Planning: Initialize Context Snapshot

    Planning --> ActionSelection: Formulate Sub-task Graph
    Planning --> Fallback: Plan Validation Rejected

    ActionSelection --> ToolExecuting: Dispatched Deterministic Tool
    ActionSelection --> Verification: Direct Synthesis / No Tool Needed
    ActionSelection --> HumanInTheLoop: High-Risk Action Policy Triggered

    HumanInTheLoop --> ToolExecuting: Approval Received via Webhook
    HumanInTheLoop --> Fallback: Rejected / Timeout Window Expired

    ToolExecuting --> Verification: Tool Execution Succeeded
    ToolExecuting --> Fallback: Circuit Breaker Open / Max Retries Met

    Verification --> ActionSelection: Contract Verified & More Steps Left
    Verification --> Completed: Goal Verified & Task Done
    Verification --> Planning: Verification Failed & Re-planning Required

    Fallback --> ActionSelection: Recovered via Alternative Path
    Fallback --> Failed: Unrecoverable Execution Failure

    Completed --> [*]
    Failed --> [*]

2. Key Elements of an Agentic State Machine

Structured State Snapshot:
Instead of appending raw message blobs, production systems manage a strongly-typed AgentState object that cleanly isolates:
- Immutable Goal: The original customer directive, locked against contextual drift.
- Plan Graph: Verified step indexes, completed tasks, and pending milestones.
- Working Scratchpad: Ephemeral variables and tool payloads needed only for the immediate step, purged after verification to safeguard the LLM's context window.
- Circuit Breaker Registry: Real-time tracking of tool invocation frequencies, latencies, and failure counts.
Transition Matrix & Guard Functions:
Every state boundary transition must be guarded by deterministic assertions:
- For example: ToolExecuting is strictly barred from leaping back to ActionSelection. It must flow through Verification.
- A transition fires only if pydantic_schema_check(output) == True and circuit_breaker.is_open() == False.
Checkpoints & Time-Travel Debugging:
Every state transition is an immutable event. State snapshots are serialized and persisted into persistent backends (PostgreSQL, SQLite, or Redis). This unlocks two critical capabilities:
- True Human-in-the-Loop: When high-stakes operations require human approval, worker threads can safely release resources. Once an approver signs off via a webhook, the execution engine rehydrates state instantly via thread_id and checkpoint_id.
- Deterministic Trajectory Replay: If an agent fails on step 8, engineers can rewind state to step 7, adjust guiding prompts or mock data, and fork execution to reproduce edge cases cleanly. For observability best practices, see Agent Observability & Debugging.

3. Tool-Calling Defense & Circuit Breakers

In a deterministic architecture, tool invocation is treated not as a trivial try...except wrapper, but as a layered defensive perimeter:

                   ┌───────────────────────────────────────────────┐
                   │               LLM Tool Call                   │
                   └───────────────────────┬───────────────────────┘
                                           ▼
                   ┌───────────────────────────────────────────────┐
                   │  Layer 1: Deterministic Schema Validation    │
                   │  (Strict Pydantic V2 / JSON Schema Check)    │
                   └───────┬───────────────────────────────┬───────┘
                           │ Valid                         │ Invalid
                           ▼                               ▼
      ┌────────────────────────────────────┐     ┌─────────────────┐
      │  Layer 2: Circuit Breaker Check    │     │ Inject Schema   │
      │  (Inspect if target tool is OPEN)  │     │ Error Prompt    │
      └───────┬────────────────────┬───────┘     └────────┬────────┘
              │ Closed (Normal)    │ Open (Tripped)       │ (Self-Correct)
              ▼                    ▼                      ▼
      ┌───────────────┐    ┌─────────────────┐   ┌─────────────────┐
      │ Execute Tool  │    │ Route to        │   │ Retry in Current│
      │ Real API Call │    │ Fallback Tool   │   │ Node (Max 2)    │
      └───────┬───────┘    └─────────────────┘   └─────────────────┘
              │ Exception
              ▼
      ┌────────────────────────────────────┐
      │ Increment Failure Counter          │
      │ If failures >= 3 -> Open Circuit   │
      └────────────────────────────────────┘

1. Strict Schema Enforcement and Bounded Self-Correction

Models frequently commit trivial formatting errors: passing timestamps as raw strings, omitting required keys, or providing floating-point values formatted with dollar signs.
Forwarding invalid parameters to downstream production databases wastes backend compute and yields obscure error messages that further mislead the model.

Architectural Best Practice:
Deploy a strict Pydantic V2 schema validation layer ahead of API execution. When validation fails, intercept the error locally within the runtime. Feed a precise correction prompt (e.g., "Field 'limit' must be an integer between 1 and 50, but received 'unlimited'. Please fix this parameter.") directly to the decision node, capping local corrections at two retries before escalating.

2. Stateful Circuit Breakers

When an external tool suffers downstream degradation or network outages:

Closed: Normal operations. Tool calls are dispatched without restriction.
Open: The tool has failed repeatedly (e.g., 3 consecutive failures). The runtime intercepts further attempts immediately, bypassing the faulty service and directing the FSM to an alternative degraded workflow (e.g., switching from real-time database queries to read-only replica caches).
Half-Open: Following a cooling window, a single probe invocation is permitted. A successful call resets the breaker to Closed; continued failure resets the backoff timer.

4. Production Implementation: A Resilient Python State Machine Engine

The following standalone Python 3.11+ implementation provides a production-grade agent state machine engine with Pydantic V2 schema validation, stateful circuit breakers, and deterministic fallback handling without heavy framework abstractions:

"""
deterministic_agent_fsm.py
Production-grade deterministic agent FSM engine featuring circuit breakers,
strict Pydantic schema validation, and isolated working scratchpads.
"""

import logging
from enum import Enum
from typing import Any, Callable, Dict, List, Optional
from pydantic import BaseModel, Field, ValidationError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("AgentFSM")


# ==========================================================
# 1. State and Data Model Definitions
# ==========================================================

class NodeState(str, Enum):
    IDLE = "IDLE"
    PLANNING = "PLANNING"
    ACTION_SELECT = "ACTION_SELECT"
    TOOL_EXECUTION = "TOOL_EXECUTION"
    VERIFICATION = "VERIFICATION"
    FALLBACK = "FALLBACK"
    TERMINAL_SUCCESS = "TERMINAL_SUCCESS"
    TERMINAL_FAILED = "TERMINAL_FAILED"


class SearchDatabaseArgs(BaseModel):
    query: str = Field(..., min_length=2, description="Target search query")
    limit: int = Field(default=5, ge=1, le=50, description="Max returned results (1-50)")


class ToolResult(BaseModel):
    success: bool
    data: Optional[Any] = None
    error_message: Optional[str] = None


class AgentMemory(BaseModel):
    user_goal: str
    current_step: int = 0
    max_steps: int = 10
    scratchpad: List[Dict[str, Any]] = Field(default_factory=list)
    final_output: Optional[str] = None
    circuit_failures: Dict[str, int] = Field(default_factory=dict)
    circuit_max_failures: int = 3


# ==========================================================
# 2. External Service Simulator with Injected Failures
# ==========================================================

class ExternalServiceSimulator:
    """Simulates an external database service to verify circuit breaker behaviors."""
    def __init__(self, failure_trigger_count: int = 3):
        self.call_count = 0
        self.failure_trigger_count = failure_trigger_count

    def search_database(self, query: str, limit: int) -> Dict[str, Any]:
        self.call_count += 1
        # Injects deliberate connection timeouts to trigger circuit tripping
        if self.call_count <= self.failure_trigger_count:
            raise ConnectionError(f"Database cluster timeout (Attempt {self.call_count})")
        return {"status": "ok", "items": [f"Result #{i} for '{query}'" for i in range(1, limit + 1)]}


# ==========================================================
# 3. Deterministic Finite State Machine Engine
# ==========================================================

class DeterministicAgentFSM:
    def __init__(self, user_goal: str, db_service: ExternalServiceSimulator):
        self.memory = AgentMemory(user_goal=user_goal)
        self.db_service = db_service
        self.current_state = NodeState.IDLE
        self.history: List[str] = []

    def transition_to(self, next_state: NodeState, reason: str = ""):
        msg = f"Transition: [{self.current_state.value}] -> [{next_state.value}]"
        if reason:
            msg += f" (Reason: {reason})"
        logger.info(msg)
        self.history.append(f"{self.current_state.value} -> {next_state.value}")
        self.current_state = next_state

    def is_circuit_open(self, tool_name: str) -> bool:
        failures = self.memory.circuit_failures.get(tool_name, 0)
        return failures >= self.memory.circuit_max_failures

    def record_tool_failure(self, tool_name: str):
        self.memory.circuit_failures[tool_name] = self.memory.circuit_failures.get(tool_name, 0) + 1
        logger.warning(f"Tool [{tool_name}] failure count: {self.memory.circuit_failures[tool_name]}/{self.memory.circuit_max_failures}")

    def reset_tool_failure(self, tool_name: str):
        if tool_name in self.memory.circuit_failures:
            self.memory.circuit_failures[tool_name] = 0

    # ---------------- Node Execution Handlers ----------------

    def step_idle(self):
        logger.info(f"Received goal: '{self.memory.user_goal}'")
        self.transition_to(NodeState.PLANNING, "Goal initialized")

    def step_planning(self):
        self.memory.current_step += 1
        logger.info("Decomposing goal into structured verifiable steps...")
        self.transition_to(NodeState.ACTION_SELECT, "Plan ready")

    def step_action_select(self):
        self.memory.current_step += 1
        if self.memory.current_step > self.memory.max_steps:
            self.transition_to(NodeState.FALLBACK, "Step budget exceeded")
            return

        # Verify circuit breaker condition
        if self.is_circuit_open("search_database"):
            logger.error("Circuit breaker is OPEN for 'search_database'. Execution intercepted.")
            self.transition_to(NodeState.FALLBACK, "Circuit breaker tripped")
            return

        # Emulate LLM tool dispatch with arguments
        mock_model_raw_args = {"query": "production_incident_report", "limit": 3}

        # Layer 1: Strict Pydantic schema validation
        try:
            validated_args = SearchDatabaseArgs(**mock_model_raw_args)
            self.memory.scratchpad.append({
                "action": "search_database",
                "args": validated_args.model_dump()
            })
            self.transition_to(NodeState.TOOL_EXECUTION, "Schema validation passed")
        except ValidationError as e:
            logger.error(f"Schema validation failed: {e.errors()}")
            self.transition_to(NodeState.PLANNING, "Schema repair requested")

    def step_tool_execution(self):
        last_action = self.memory.scratchpad[-1]
        tool_name = last_action["action"]
        args = last_action["args"]

        logger.info(f"Executing tool '{tool_name}' with args: {args}")
        try:
            raw_response = self.db_service.search_database(args["query"], args["limit"])
            self.reset_tool_failure(tool_name)
            last_action["result"] = ToolResult(success=True, data=raw_response)
            self.transition_to(NodeState.VERIFICATION, "Tool executed successfully")
        except Exception as err:
            logger.error(f"Execution error: {err}")
            self.record_tool_failure(tool_name)
            last_action["result"] = ToolResult(success=False, error_message=str(err))

            if self.is_circuit_open(tool_name):
                self.transition_to(NodeState.FALLBACK, f"Tool [{tool_name}] reached max failure threshold")
            else:
                self.transition_to(NodeState.ACTION_SELECT, "Retry with remaining budget")

    def step_verification(self):
        last_action = self.memory.scratchpad[-1]
        result: ToolResult = last_action.get("result")

        if result and result.success:
            logger.info("Verification passed: Structured data confirmed.")
            self.memory.final_output = f"Successfully fetched {len(result.data['items'])} items."
            self.transition_to(NodeState.TERMINAL_SUCCESS, "Goal completed")
        else:
            logger.warning("Verification failed: Data incomplete.")
            self.transition_to(NodeState.PLANNING, "Re-planning required")

    def step_fallback(self):
        logger.info("Entering FALLBACK node. Triggering graceful degradation...")
        self.memory.final_output = "Gracefully degraded: Returning cached incident summaries."
        self.transition_to(NodeState.TERMINAL_SUCCESS, "Recovered via Fallback degradation")

    # ---------------- FSM Execution Loop ----------------

    def run(self) -> str:
        handlers: Dict[NodeState, Callable[[], None]] = {
            NodeState.IDLE: self.step_idle,
            NodeState.PLANNING: self.step_planning,
            NodeState.ACTION_SELECT: self.step_action_select,
            NodeState.TOOL_EXECUTION: self.step_tool_execution,
            NodeState.VERIFICATION: self.step_verification,
            NodeState.FALLBACK: self.step_fallback,
        }

        while self.current_state not in (NodeState.TERMINAL_SUCCESS, NodeState.TERMINAL_FAILED):
            handler = handlers.get(self.current_state)
            if not handler:
                raise RuntimeError(f"Unhandled FSM state: {self.current_state}")
            handler()

        logger.info(f"FSM Terminated at [{self.current_state.value}]. Output: {self.memory.final_output}")
        return self.memory.final_output


if __name__ == "__main__":
    print("\n--- [Test Scenario: Flaky Service Tripping Circuit Breaker to Fallback] ---")
    failing_service = ExternalServiceSimulator(failure_trigger_count=3)
    agent = DeterministicAgentFSM(
        user_goal="Query latest production diagnosis report",
        db_service=failing_service
    )
    result = agent.run()

    print("\n--- Execution Trajectory Trace ---")
    for idx, transition in enumerate(agent.history, 1):
        print(f"Step {idx:02d}: {transition}")

Execution Log Trace

Executing the script yields a clean, audited, and deterministic transition trace:

2026-09-20 10:15:01 [INFO] Received goal: 'Query latest production diagnosis report'
2026-09-20 10:15:01 [INFO] Transition: [IDLE] -> [PLANNING] (Reason: Goal initialized)
2026-09-20 10:15:01 [INFO] Decomposing goal into structured verifiable steps...
2026-09-20 10:15:01 [INFO] Transition: [PLANNING] -> [ACTION_SELECT] (Reason: Plan ready)
2026-09-20 10:15:01 [INFO] Transition: [ACTION_SELECT] -> [TOOL_EXECUTION] (Reason: Schema validation passed)
2026-09-20 10:15:01 [INFO] Executing tool 'search_database' with args: {'query': 'production_incident_report', 'limit': 3}
2026-09-20 10:15:01 [ERROR] Execution error: Database cluster timeout (Attempt 1)
2026-09-20 10:15:01 [WARNING] Tool [search_database] failure count: 1/3
2026-09-20 10:15:01 [INFO] Transition: [TOOL_EXECUTION] -> [ACTION_SELECT] (Reason: Retry with remaining budget)
2026-09-20 10:15:01 [INFO] Transition: [ACTION_SELECT] -> [TOOL_EXECUTION] (Reason: Schema validation passed)
2026-09-20 10:15:01 [ERROR] Execution error: Database cluster timeout (Attempt 2)
2026-09-20 10:15:01 [WARNING] Tool [search_database] failure count: 2/3
2026-09-20 10:15:01 [INFO] Transition: [TOOL_EXECUTION] -> [ACTION_SELECT] (Reason: Retry with remaining budget)
2026-09-20 10:15:01 [INFO] Transition: [ACTION_SELECT] -> [TOOL_EXECUTION] (Reason: Schema validation passed)
2026-09-20 10:15:01 [ERROR] Execution error: Database cluster timeout (Attempt 3)
2026-09-20 10:15:01 [WARNING] Tool [search_database] failure count: 3/3
2026-09-20 10:15:01 [INFO] Transition: [TOOL_EXECUTION] -> [FALLBACK] (Reason: Tool [search_database] reached max failure threshold)
2026-09-20 10:15:01 [INFO] Entering FALLBACK node. Triggering graceful degradation...
2026-09-20 10:15:01 [INFO] Transition: [FALLBACK] -> [TERMINAL_SUCCESS] (Reason: Recovered via Fallback degradation)

In a standard ReAct loop without circuit breakers, the agent would continue churning through attempts until hitting gateway timeouts. In contrast, the FSM trips immediately upon hitting the failure threshold, routing execution safely to the FALLBACK degradation node.

5. Architectural Memory Isolation: Scratchpads vs Checkpointers

A critical error in enterprise agent design is appending every raw tool payload to a singular conversational history. In state machine architecture, memory must be stratified:

Memory Tier	Stored Contents	Lifecycle	Context Window Impact
Working Scratchpad	Raw intermediate tool outputs, scraped HTML, unparsed JSON	Ephemeral (alive only during node execution; discarded or distilled upon transition)	Injected only into localized single-step LLM calls
State Checkpointer	Sub-task progress, verified plan graphs, entity state dictionaries	Persistent across task execution (persisted to PostgreSQL / Redis)	Distilled into compact summaries during re-planning
Episodic Long-Term Memory	Cross-session user preferences, learned historical workflows, corporate knowledge bases	Permanent storage across user sessions (Vector DBs / Knowledge Graphs)	Injected dynamically via Top-K semantic retrieval

For deep dives into external retrieval and context hygiene, explore our practical guide on Enterprise RAG in Practice.

6. The Production AI Agent Architect Handbook Roadmap

Deterministic control flows form the first essential pillar of production-grade agent engineering. Once the deterministic skeleton is locked in, advanced reasoning capabilities can be deployed safely without jeopardizing business reliability.

Explore the complete curriculum across the 10 chapters of this handbook:

Part 1 (Current) · Agentic Loops & State Machine Design: From ReAct to Deterministic Control Flows Foundational state machine topologies, transition guards, and circuit breaker patterns.
Part 2 · Building an AI Agent from Scratch: ReAct Loops & Architecture Test-driven engineering for autonomous agent cores and tool orchestrators.
Part 3 · 7 Production Runtime Practices for AI Agents Production runtime patterns, timeout debouncing, and concurrency isolation.
Part 4 · Deep Dive into Model Context Protocol (MCP): The USB-C for AI Anthropic's standardized protocol for custom tool servers and secure enterprise governance.
Part 5 · Deep Dive into Skills: Giving AI Coding Agents a Specialized Brain Progressive context loading mechanisms and modular coding agent architectures.
Part 6 · Under the Hood of Browser-use: DOM Distillation & Vision Grounding Crossing from API calls to visual web automation with 100k+ star architecture insights.
Part 7 · Evolving Models at Runtime: From Basic Reflection to MCTS-based Test-Time Compute Reflexion loops, iterative verification, and Monte Carlo Tree Search at inference time.
Part 8 · 2026 AI Paradigm Shift: Distributed Agent Orchestration & Evals Multi-agent swarms, combating compounding error rates, and automated benchmark evals.
Part 9 · Agent Observability & Debugging: From Black Box to White Box OpenTelemetry, LangSmith, and trajectory replay for complete execution transparency.
Part 10 · How Environment Scaling Reshapes Autonomous Agents: Sandboxing & RL Post-training environment exploration, zero-escape container sandboxing, and benchmark dominance.

Frequently Asked Questions (FAQ)

Q1: Given that modern LLMs exhibit impressive autonomous reasoning, why constrain them with rigid state machines rather than using fully autonomous loops?

The reasoning capability of LLMs is mathematically rooted in probabilistic token sampling, which cannot provide deterministic safety guarantees. In production environments, tasks frequently perform mutating state modifications across enterprise backends (e.g., executing debit payments, writing cloud configs, or deleting records). If an LLM is granted unconstrained control over state flow, unexpected network jitter, boundary inputs, or prompt injection can trap the model in repetitive oscillations or bypass mandatory security verification steps. A Finite State Machine (FSM) establishes code-level immutable edges for business policies while delegating flexible semantic reasoning to nodes inside safe sandboxes.

Q2: How does a state machine prevent context window explosion and attention decay across long-running multi-step workflows?

Through strict separation between ephemeral scratchpads and persistent checkpointers. In an FSM, large raw payloads produced by tool executions (such as 100KB database outputs or raw web pages) exist only within the current node's localized working scratchpad. Before transitioning to downstream nodes, a verification step condenses raw data into concise typed summaries or entity dictionaries stored in the state checkpointer. Detailed raw traces are offloaded directly to asynchronous telemetry collectors (e.g., Langfuse / OpenTelemetry), ensuring the active context window remains lean and constant throughout multi-step execution.

Q3: How should Human-in-the-Loop (HITL) workflows be architected within an agent state machine without blocking backend workers?

Through persistent checkpointer serialization coupled with asynchronous event webhooks. When an FSM routes to a HumanInTheLoop node for high-risk approvals, the engine serializes the complete AgentState snapshot to a persistent store (e.g., PostgreSQL or Redis) tagged with a session_id and checkpoint_id. The active worker thread or coroutine immediately releases its resources back to the pool. When a human reviewer approves or rejects the action via Slack, enterprise messaging, or an admin portal, an incoming webhook retrieves the snapshot by checkpoint_id, restores the exact execution context on an available worker, and resumes execution seamlessly without state loss.

Originally published at Nobita Talks AI on blog.llmgo.top.

Under the Hood of Browser-use (100k+ Stars): DOM Tree Distillation, Vision Grounding, and Production Web Agents

ifnodoraemon — Sun, 20 Sep 2026 05:05:08 +0000

Introduction: The Death of Fragile RPA and Selector-Based Automation

For more than a decade, web automation—spanning web scraping, QA regression testing, and Robotic Process Automation (RPA)—has been plagued by a fundamental failure mode: brittle XPath and CSS selectors.

Whenever frontend teams refactor component trees, re-nest a container div, or adopt utility-first CSS frameworks with randomized hash classes (e.g. class="flex_a8f9z bg-blue_39kd"), carefully maintained Playwright or Selenium scripts break without warning.

Entering 2026, this fragile paradigm has been replaced by autonomous web agents. Pioneered by projects like Browser-use, which quickly surpassed 100,000 stars on GitHub, automation has shifted toward multimodal cognitive interaction. Instead of matching brittle DOM paths, the model navigates the web like a human: perceiving rendered screenshots with its eyes, understanding interactive semantics with its brain, planning multi-step actions, and self-healing when encountering unexpected modals or bot challenges.

graph LR
    subgraph Fragile Traditional Automation
        A1["Dynamic Target Webpage"] --> A2["Hardcoded XPath/CSS Selectors"] --> A3["Frontend Refactor (Class Changed)"] --> A4["❌ Script Throws Execution Error"]
    end
    subgraph Cognitive Autonomous Browser-use
        B1["Dynamic Target Webpage"] --> B2["DOM Pruning + Set-of-Mark (SoM) Overlays"] --> B3["Multimodal LLM Reasoning Engine"] --> B4["Dynamic Adaptive Actions"] --> B5["✅ Resilient Task Completion"]
    end

This article deconstructs the architectural foundations of Browser-use, explaining how it mitigates token explosion, eliminates visual coordinate drift, evades bot detection, and executes multi-step enterprise workflows.

I. Dual-Modal Perception Core: DOM Tree Distillation & Set-of-Mark Grounding

Passing raw modern Single Page Application (SPA) HTML directly into an LLM context window is disastrous: pages often span megabytes of minified code, consuming tens of thousands of tokens and diluting the model with extraneous script, style, and svg metadata.

Browser-use resolves this with an efficient dual-modal perception pipeline:

graph TD
    A["Webpage Rendering (Playwright Chromium)"] --> B["Dual Extraction: DOM Snapshot + High-Res Viewport Screenshot"]

    subgraph Text Modality: Heuristic DOM Tree Distillation
        B --> C1["Strip scripts, styles, SVGs, and hidden elements"]
        C1 --> C2["Extract interactive nodes (button, input, a, select)"]
        C2 --> C3["Compute element BoundingBoxes & viewport visibility"]
        C3 --> C4["Distill into lightweight semantic tree (~5KB - 15KB)"]
    end

    subgraph Vision Modality: Set-of-Mark (SoM) Coordinate Grounding
        B --> D1["Render bounding boxes on high-res screenshot"]
        D1 --> D2["Assign unique numerical badge IDs to interactive elements"]
        D2 --> D3["Generate Set-of-Mark (SoM) visual screenshot"]
    end

    C4 --> E["Multimodal LLM Unified Context"]
    D3 --> E

1. Heuristic DOM Tree Pruning

Browser-use injects a specialized traversal script into the browser runtime to prune non-essential DOM structures:

Visibility Filtering: Evaluates computed styles (display === 'none', visibility === 'hidden', opacity === '0') and discards nodes outside the active viewport.
Metadata Stripping: Purges all <script>, <style>, <link>, and <meta> tags alongside SVG paths.
Semantic Flattening: Collapses non-semantic wrapper <div> chains, preserving only nodes that contain meaningful text or interactive accessibility attributes (aria-label, placeholder, role, href).
Result: Raw HTML trees shrinking from 2MB+ down to an information-dense representation of 1,500 to 3,000 tokens. For strategies on managing runtime context budgets, review our Context Engineering Guide.

2. Set-of-Mark (SoM) Visual Grounding

Predicting exact pixel coordinates (e.g. click at x=1240, y=850) often fails due to display scaling differences and viewport misalignments.

Browser-use implements Set-of-Mark (SoM) visual grounding:

The injected JavaScript retrieves the bounding box for every interactive candidate.
It paints an overlay of brightly colored bounding boxes containing numerical badges ([1], [2], [15]) directly onto the page.
It takes a clean screenshot of this annotated state.
The Model's Action Space is Simplified: The model does not calculate coordinates; it outputs high-level actions referencing IDs, such as click_element(index=14) or input_text(index=3, text="admin@company.com"). This increases action execution precision beyond 95%.

II. Agent Cognitive Loop & State Machine Resilience

Enterprise web automation rarely involves single-shot actions; it demands workflows spanning 10 to 30 sequential steps. Browser-use manages this via an Observe-Reason-Plan-Act-Verify (O-P-A-V) state machine:

┌─────────────────────────────────────────────────────────────┐
│                 Browser-use Cognitive Loop                  │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
 1. Observe   ───► Prune DOM and capture SoM-annotated screenshot
                             │
                             ▼
 2. Reason    ───► Evaluate user goal vs. current interactive state
                             │
                             ▼
 3. Plan      ───► Select atomic action: Click / Type / Scroll / Tab
                             │
                             ▼
 4. Act       ───► Dispatch simulated input via Playwright CDP
                             │
                             ▼
 5. Verify    ───► Wait for network idle & DOM mutation; verify state
                             │
                             ├─► [Success] Proceed to next observation loop
                             └─► [Failure] Trigger self-healing retry or fallback

1. Dynamic Viewport Exploration and Scrolling

When information resides below the fold or inside nested containers, the agent utilizes dedicated scroll primitives:

scroll_down(amount=500): Scrolls the viewport and re-evaluates the fresh SoM layout.
Maintains a lightweight historical trajectory of scrolled coordinates to avoid cycling endlessly between page extremes. For details on self-correction loops, see our guide on Agent Reflection and Self-Correction.

2. State Stagnation Detection & Self-Healing

If an agent attempts three consecutive clicks on an element without altering DOM topology or visual layout, the built-in watchdog triggers recovery:

Forces a clean page reload;
Injects contextual feedback prompting the model: "Previous click failed to trigger a state mutation. Inspect whether a blocking modal overlay exists or if scrolling is required."

III. Production Hardening: Anti-Bot Evasion & Authentication

Moving Browser-use into unattended 24/7 enterprise production environments requires handling real-world web defenses:

1. WAF & Fingerprint Detection Evasion

Standard Headless Chrome exposes obvious automation artifacts (navigator.webdriver = true, missing WebGL vendor strings, static screen dimensions), leading to immediate blocks by Cloudflare or DataDome.

Production Hardening Checklist:

Deploy playwright-stealth to scrub known automation properties;
Randomize User-Agent headers, viewport dimensions, and hardware concurrency metrics;
Add randomized micro-jitters to typing events and simulate human-like Bezier mouse curves.

2. CAPTCHAs & Two-Factor Authentication (Human-in-the-Loop)

Attempting to fully automate adversarial 3D rotation or visual puzzle CAPTCHAs in production introduces fragility.

Best Practice: Human-in-the-Loop Handover.
When an agent detects a blocking challenge, it suspends its automated loop and issues a real-time notification (via Webhook, Slack, or Teams). A human operator opens a remote visual debug session via the Chrome DevTools Protocol (CDP), resolves the challenge within 30 seconds, and signals the agent to resume execution.

IV. Hands-on Implementation: Building an Autonomous Financial Research Agent

Here is a complete, runnable script configuring Browser-use with multimodal models to execute a multi-step financial report download workflow.

1. Environment Installation

pip install browser-use playwright langchain-openai
playwright install chromium

2. Complete Python Implementation

import os
import asyncio
from browser_use import Agent, Browser, BrowserConfig
from browser_use.browser.context import BrowserContextConfig
from langchain_openai import ChatOpenAI

# 1. Configure the multimodal vision backbone
# For production, utilize models with strong spatial grounding (GPT-4o, Claude 3.7 Sonnet, or Qwen2.5-VL)
llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0.0,
    api_key=os.getenv("OPENAI_API_KEY")
)

# 2. Production browser resilience configuration
browser_config = BrowserConfig(
    headless=False,            # Set to True for unattended production deployment
    disable_security=True,     # Disregard self-signed intranet SSL certificates
    extra_chromium_args=[
        "--disable-blink-features=AutomationControlled",  # Scrub webdriver automation flag
        "--no-sandbox",
        "--window-size=1920,1080"
    ]
)

# 3. Session isolation and recording configuration
context_config = BrowserContextConfig(
    viewport={"width": 1920, "height": 1080},
    save_recording_path="./recordings",  # Record video artifacts for auditing and triage
    locale="en-US"
)

async def run_enterprise_workflow():
    browser = Browser(config=browser_config)
    context = await browser.new_context(config=context_config)

    # 4. Define high-level business objective
    task_prompt = """
    1. Navigate to https://example-finance-portal.com/login
    2. If a login form is displayed, enter 'finance_bot' as username, 'Secr3t_Pass!' as password, and click Submit
    3. Once logged in, locate and click 'Financial Reconciliation' in the sidebar
    4. Set the date range filter to '2026 Q2', then click 'Export Summary CSV'
    5. Wait for the download to complete and confirm that a success modal appears
    """

    # 5. Initialize the Browser-use Agent
    agent = Agent(
        task=task_prompt,
        llm=llm,
        browser_context=context,
        use_vision=True,                # Enable Set-of-Mark visual grounding
        max_failures=3,                 # Maximum retry attempts per step
        max_actions_per_step=4          # Allow compound actions per turn to reduce latency
    )

    print("🚀 Launching enterprise autonomous Browser-use agent...")
    history = await agent.run(max_steps=25)

    print("\n✅ Execution Summary:")
    print(history.final_result())

    await context.close()
    await browser.close()

if __name__ == "__main__":
    asyncio.run(run_enterprise_workflow())

By eliminating hardcoded selectors, the agent remains functional even when frontend teams update layout classes. For debugging and session logging, review our Agent Observability and Debugging Guide.

V. Enterprise Architecture & Cost Trade-offs

Autonomous web agents do not completely supplant direct API calls. Production systems should adopt a layered automation hierarchy:

Automation Layer	Ideal Workload	Execution Latency	Cost per Step	Reliability
Direct REST/GraphQL APIs	Standardized, documented internal endpoints	Sub-100ms	Free	High (Schema Contract)
Deterministic Playwright Scripts	High-volume, static regression smoke tests	Sub-second	Free	Medium (Breaks on UI updates)
Browser-use Vision Agents	Legacy systems without APIs, anti-bot sites, exploratory workflows	~3s per step	$0.01 - $0.05 per step (Token-based)	High (Adaptive self-healing)

Frequently Asked Questions (FAQ)

Q1: Does capturing screenshots on every step cause token costs to explode?

No. Browser-use automatically compresses and resizes screenshots (typically capping resolution at 1280px width), keeping per-image consumption within a few hundred tokens. Combined with distilled DOM pruning, the average per-step cost remains between 800 and 2,500 tokens. For high-volume private workflows, hosting open-weight models (such as Qwen2.5-VL-72B via vLLM Serving) reduces per-token marginal costs to near zero.

Q2: How does the agent track state across multi-tab popups?

Browser-use maintains an integrated tab manager. When a click triggers window.open or a popup, Playwright catches the context.on('page') event and injects the active tab list (available_tabs: [Tab 0: Home, Tab 1: Checkout]) into the prompt state. The model issues switch_tab(tab_id=1) commands while maintaining unified execution history in the primary controller.

Q3: Can Browser-use run in headless containerized cloud environments?

Yes. In production Kubernetes clusters, the standard deployment pattern packages the agent inside a lightweight Linux container utilizing Xvfb (X Virtual Framebuffer) to emulate a display. This enables headless execution while preserving full screenshot and Set-of-Mark visual grounding capabilities.

Originally published at Nobita Talks AI on blog.llmgo.top.

vLLM Online Inference in Production: From Architecture to Token Billing

ifnodoraemon — Sun, 20 Sep 2026 05:04:59 +0000

Introduction: Why vLLM?

In 2026, if you need to deploy an LLM inference service in production — whether it's an internal AI assistant or a commercial API platform — you'll almost certainly encounter vLLM.

Born in UC Berkeley's Sky Computing Lab and published at SOSP 2023, vLLM has become one of the most active open-source inference engines with 2,000+ contributors. Its motto is straightforward: "Easy, fast, and cheap LLM serving for everyone."

But what exactly makes vLLM fast? How do you monitor and optimize cache hit rates? What should --max-num-seqs be set to? Why is token billing such a deep trap in self-hosted inference? This article covers architecture, caching mechanisms, scheduling algorithms, deployment, performance tuning, and token billing — everything you need for production-ready vLLM.

Layer 1: Understanding vLLM's Core Engine in 5 Minutes

LLM inference fundamentally splits into two distinct phases:

Prefill: Process the entire user input at once, computing attention for all tokens. This is compute-bound — GPU cores are maxing out on matrix multiplications.
Decode: Generate output tokens one by one, each depending on the KV Cache of all preceding tokens. This is memory-bandwidth-bound.

Understanding this fundamental difference is the key to understanding every vLLM optimization.

Traditional frameworks have two fatal problems:

Memory waste: Each request's KV Cache requires pre-allocated contiguous GPU memory, even if only 10% is used.
Poor concurrency: Static batching waits for the slowest request to finish before processing the next batch.

1. PagedAttention: Virtual Memory for GPU

vLLM's most groundbreaking innovation, borrowing the virtual memory paging concept from operating systems:

Traditional: Pre-allocate contiguous memory per request (like MS-DOS real mode)
  Request A: [████████████░░░░░░░░] ← 60% wasted
  Request B: [██████░░░░░░░░░░░░░░] ← 70% wasted
  Request C: "Sorry, out of memory. Please queue."

PagedAttention: Slice memory into fixed-size pages (Blocks), allocate on demand
  Physical blocks:     [P1][P2][P3][P4][P5][P6][P7][P8]
  Request A page table: P1 → P3 → P5 (use only what you need, non-contiguous OK)
  Request B page table: P2 → P4
  Request C page table: P6 → P7 (previously couldn't fit, now easily admitted)
  Free pool:           P8 (ready for new requests or existing request growth)

Key insight: Pages don't need to be physically contiguous. vLLM maintains a Block Table for mapping, achieving three breakthroughs:

Near-zero waste: KV Cache memory waste drops from 60-80% to <4% (only the last block's internal fragmentation).
2-4x concurrency: Same GPU handles 2-4x more concurrent requests.
Shared reuse: Requests sharing the same prefix (e.g., system prompt) can point to the same physical pages.

2. Continuous Batching

Traditional inference is like a restaurant requiring all guests to arrive, order, and leave simultaneously. vLLM introduces iteration-level scheduling:

Timeline:
  t=0  [Req A-decode] [Req B-decode] [Req C-prefill] [          ]
  t=1  [Req A-decode] [Req B-done✓]  [Req C-decode]  [Req D-new!]
  t=2  [Req A-decode] [Req D-prefill] [Req C-decode]  [          ]
  t=3  [Req A-done✓]  [Req D-decode]  [Req C-done✓]  [Req E-new!]

Request finishes → immediately removed. New request arrives → immediately inserted. GPU stays fully utilized at all times.

3. Chunked Prefill — The Latency Lifesaver for Online Serving

A hidden danger in continuous batching: a 10,000-token document will monopolize the GPU for hundreds of milliseconds during prefill, stalling all other decoding requests.

Chunked prefill divides long prompts into smaller chunks:

Scheduler maintains a "token budget" per iteration (e.g., max_num_batched_tokens = 8192)

Round 1: [Req A prefix tokens 0-2047]  + [Req B decode 1 token] + [Req C decode 1 token]
Round 2: [Req A prefix tokens 2048-4095] + [Req B decode 1 token] + [Req C decode 1 token]
...
Round 5: [Req A remaining prefix tokens] + [Req B decode 1 token] + [Req C decode 1 token]
Round 6: [Req A starts decoding!] + [Req B decode 1 token] + [Req C decode 1 token]

Core scheduling policy: vLLM V1 engine (v0.29.0) prioritizes active decode requests first, then uses remaining budget for new prefill chunks. Users B and C barely notice User A's massive input.

Must-enable for online services: --enable-chunked-prefill.

Layer 2 Deep Dive: APC Prefix Caching — Understanding It Pays Off

If PagedAttention solves "how to allocate memory," Automatic Prefix Caching (APC) solves "how to reuse what's already been computed."

Why APC Is Critical for Online Services

Real-world online inference requests typically look like:

Req 1: [System Prompt (800T) | Multi-turn History (2000T) | New Message (50T)]
Req 2: [System Prompt (800T) | Multi-turn History (2000T) | New Message (30T)]
Req 3: [System Prompt (800T) | Different User Chat (500T) | New Message (80T)]

Requests 1 and 2 share 2,800 tokens of prefix! Request 3 shares 800 tokens of system prompt. Recomputing from scratch every time is enormously wasteful.

How APC Works: Block Hashing + Global Hash Table + LRU Eviction

Step 1: Block Hashing

vLLM partitions all token sequences into fixed-size blocks (default: 16 tokens). Each block is uniquely identified by a chained hash:

Hash calculation: hash(parent_block_hash + current_block_tokens)

Example: System prompt "You are a helpful AI assistant..." (800 tokens)
  Block_0: hash(NULL + tokens[0:16])       = 0xA1B2C3...
  Block_1: hash(0xA1B2C3 + tokens[16:32])  = 0xD4E5F6...
  Block_2: hash(0xD4E5F6 + tokens[32:48])  = 0x789ABC...
  ...     (50 blocks total)

Why chained hashing? The same 16 tokens at different context positions produce entirely different attention outputs. Chain hashing ensures only identical prefixes match.

Step 2: Global Hash Table Lookup

vLLM maintains a global HashMap<BlockHash, PhysicalBlock>. When a new request arrives:

New request: [System Prompt | User Chat | New Message]
1. Scheduler computes hash for each block
2. Queries global hash table one by one:
   Block_0 (0xA1B2C3) → HIT! Reference physical Block #42
   Block_1 (0xD4E5F6) → HIT! Reference physical Block #43
   ... (49 more hits)
   Block_50 (0xNEW001) → MISS. New message content, needs computation
3. Result: 50 blocks reused, only compute from Block_50 onwards
   → Prefill computation reduced by 98%!

Step 3: LRU Eviction Policy

GPU memory is finite. vLLM uses reference counting + LRU to manage the cache:

Each physical block has two key attributes:
  - ref_count: how many active requests reference this block
  - last_access_time: timestamp of last access

Eviction rules:
  1. Blocks with ref_count > 0 are NEVER evicted (actively in use!)
  2. Blocks with ref_count = 0 enter the "candidate pool"
  3. When memory runs low, evict the block with the oldest last_access_time
  4. Remove its hash from the global table

Four Key APC Use Cases

Scenario	Reusable Prefix	Performance Gain
Multi-turn chat	System prompt + chat history	Later turns approach zero prefill time
Document QA (RAG)	Thousands of document tokens	Multiple queries on same doc = nearly free
Code completion	Existing file content	Incremental computation per keystroke
Template batch processing	Identical instruction prefix	1000 same-template requests compute prefix once

APC Limitations

Output >> Input: Prefill is a small fraction of total latency
Every request is unique: No shared prefixes, no cache hits
Minor miss overhead: Hash computation costs CPU time (usually negligible)

Key config: --enable-prefix-caching (default in vLLM V1 engine / v0.29.0).

Getting Cache Hit Data in API Responses

vLLM can tell you exactly how many tokens hit the cache in each response. Just add one startup parameter:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --enable-prefix-caching \
  --enable-prompt-tokens-details    # ← Key! Adds cached_tokens to usage

The usage field will now include prompt_tokens_details:

{
  "usage": {
    "prompt_tokens": 3000,
    "completion_tokens": 423,
    "total_tokens": 3423,
    "prompt_tokens_details": {
      "cached_tokens": 2800
    }
  }
}

What this means: Of 3000 prompt tokens, 2800 were reused from cache, only 200 needed GPU compute. Your cost is 6.7% of list price, but you bill the user for all 3000 tokens.

Critical for billing: Your middleware should record both prompt_tokens (user billing basis) and cached_tokens (your actual cost basis). The gap is your profit.

Cache Hit Rate Monitoring: Prometheus Metrics

vLLM V1 engine (v0.29.0) uses Counter-based metrics (replacing the deprecated gpu_prefix_cache_hit_rate Gauge):

# V1 prefix cache metrics (Counter type, precise and reliable)
vllm_prefix_cache_hits              # Cache hit tokens total (Counter)
vllm_prefix_cache_queries           # Cache query tokens total (Counter)

# PromQL for real-time hit rate (use this in Grafana)
(rate(vllm_prefix_cache_hits[5m]) / rate(vllm_prefix_cache_queries[5m])) * 100

Healthy cache hit rate benchmarks:

Scenario	Expected Hit Rate	If Lower, Check
Multi-turn chat (same session)	80%-95%	Dynamic content inserted in prefix
RAG document QA	60%-85%	Document diversity too high, or cache pool too small
Template batch processing	90%+	Template not at the front of prompt
Completely random requests	<10%	Normal — APC doesn’t help here

Best Practices for Maximizing Cache Hits

Static content first: Place system instructions and documents at the beginning of prompts, dynamic content (user messages) after.
Avoid dynamic elements in prefixes: e.g., "Current time is 2026-04-14 10:00" — a time change invalidates the entire cache chain. Put timestamps at the end.
Increase --gpu-memory-utilization: More VRAM = larger cache pool = lower LRU eviction rate = higher hit rates.
Maintain session continuity: Let vLLM naturally accumulate cache across turns.
Monitor cached_tokens / prompt_tokens ratio: This is your "profit margin". Analyze by user/scenario to identify low-hit-rate patterns.

Layer 3: OpenAI-Compatible API & Advanced Capabilities

vLLM natively provides an OpenAI-compatible API — your application code stays the same, just change the base_url.

Quick Start

pip install vllm

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --dtype auto \
  --api-key your-secret-key \
  --port 8000 \
  --enable-prefix-caching

Supported Endpoints

Endpoint	Path	Purpose
Chat Completions	`/v1/chat/completions`	Conversational AI
Completions	`/v1/completions`	Text completion
Responses	`/v1/responses`	OpenAI Responses API
Embeddings	`/v1/embeddings`	Vector embeddings
Models	`/v1/models`	List models
Tokenizer	`/tokenize` / `/detokenize`	Token encode/decode (billing helper)
Transcriptions	`/v1/audio/transcriptions`	Speech-to-text
Realtime	`/v1/realtime`	Real-time voice (WebSocket)

Structured Output & Guided Decoding

Online services often require strictly formatted JSON output. vLLM has built-in Guided Decoding support:

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-72B-Instruct",
    messages=[{"role": "user", "content": "Extract names and locations"}],
    extra_body={
        "guided_json": {
            "type": "object",
            "properties": {
                "names": {"type": "array", "items": {"type": "string"}},
                "locations": {"type": "array", "items": {"type": "string"}}
            },
            "required": ["names", "locations"]
        }
    },
)

Four constraint formats: guided_json, guided_regex, guided_choice, guided_grammar. Powered by XGrammar (pushdown automata-based), with <5% latency impact.

Multi-LoRA Serving

Serve multiple fine-tuned adapters on a single base model:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --enable-lora \
  --max-loras 8 \
  --lora-modules customer-a=/path/to/lora-a customer-b=/path/to/lora-b

Uses batched GEMM to dynamically group sequences using different adapters. Each LoRA adapter is typically only tens of MB vs. hundreds of GB for the base model.

Docker Production Deployment

docker run -d \
  --name vllm-server \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v /path/to/models:/models \
  -e VLLM_API_KEY=your-secret-key \
  -e PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus \
  vllm/vllm-openai:v0.29.0 \
  --model /models/Qwen2.5-72B-Instruct \
  --dtype auto \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-prompt-tokens-details    # Expose cached_tokens for billing

⚠️ Always use --ipc=host: vLLM uses PyTorch shared memory for multi-process communication. Without this flag, you'll get errors inside containers. This is the #1 beginner mistake.

Layer 4: Performance Tuning

Core Metrics for Online Services

Metric	Full Name	Meaning	Goal
TTFT	Time to First Token	Time from request to first token	Lower → "responsive"
ITL	Inter-Token Latency	Gap between consecutive tokens	Lower → "smooth typing"
Throughput	tokens/s	Total tokens processed per second	Higher → lower cost per token

Core tension: Higher throughput and lower latency are naturally opposed. Online services must find the sweet spot.

Complete Tuning Guide

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --gpu-memory-utilization 0.90 \     # [1]
  --max-model-len 8192 \              # [2] THE MOST CRITICAL PARAMETER
  --max-num-seqs 256 \                # [3]
  --max-num-batched-tokens 16384 \    # [4]
  --enable-chunked-prefill \          # [5]
  --enable-prefix-caching \           # [6]
  --quantization fp8 \                # [7]
  --kv-cache-dtype fp8_e5m2 \         # [8]
  --tensor-parallel-size 2 \          # [9]
  --num-speculative-tokens 5 \        # [10]
  --speculative-model Qwen/Qwen2.5-1.5B-Instruct

[1] `--gpu-memory-utilization`

0.70: 30% wasted → tiny APC cache pool → hit rate plummets
0.98: almost no buffer → OOM crash during spikes → all in-flight requests lost
0.90~0.95: sweet spot for production

[2] `--max-model-len` — THE #1 PERFORMANCE KILLER

vLLM pre-calculates KV Cache space based on this value. Many beginners use the model default (e.g., Qwen2.5's 131,072).

Default 131072 → KV Cache reserves 30x more space → concurrency drops from 50 to 2
Set to actual need 8192 → KV Cache right-sized → concurrency restored

This single parameter change can be more impactful than all other optimizations combined.

[3-4] Concurrency & Token Budget

Online chat (latency-first): max-num-seqs 64-256, max-num-batched-tokens 8192-16384
Batch processing (throughput-first): max-num-seqs 2048-4096, max-num-batched-tokens 32768

[7-8] Quantization

GPU	Recommended	Accuracy Loss	Effect
H100/H800	FP8 (`--quantization fp8`)	<1%	~1.5x throughput, zero-calibration
A100/L40S	INT8 GPTQ/AWQ	≈0%	2x memory reduction
Blackwell B200	NVFP4	≤1%	2x throughput vs FP8

KV Cache Quantization (--kv-cache-dtype fp8_e5m2): Halves KV Cache memory with minimal quality impact, especially impactful for long-context (8K-128K) scenarios.

[9] Tensor Parallelism

Common mistake: 8 GPUs → TP=8 → worse throughput than TP=4

Reason: AllReduce communication overhead can eat 40% of added compute

Correct strategy:
  1. Use minimum TP to fit the model (70B FP8 on 2x H100 → TP=2)
  2. Use extra GPUs for more replicas + load balancer
  Result: 2 instances × TP=2 often 30-50% higher throughput than 1 instance × TP=4

Advanced: Check GPU topology with nvidia-smi topo -m. TP GPUs should be NVLink-connected, not PCIe-bridged (5-10x latency difference).

[10] Speculative Decoding

Low concurrency (<10 QPS): 2-5x latency improvement
High concurrency (>100 QPS): diminishing returns (GPU already saturated)

Best for: real-time chat, internal tools, latency-sensitive Agent calls
Not for: high-concurrency API services

Layer 5: Disaggregated Serving — Architecture for Scale

At hundreds to thousands of QPS, Prefill and Decode interfere with each other:

                    ┌──────────────────┐
                    │   Request Router  │
                    └──────┬───────────┘
              ┌────────────┼────────────┐
              ▼                         ▼
    ┌──────────────────┐     ┌──────────────────┐
    │  Prefill Cluster  │     │  Decode Cluster   │
    │  (Compute GPUs)   │     │  (Bandwidth GPUs) │
    │  H100 SXM / B200  │     │  L40S / A100      │
    └────────┬─────────┘     └────────▲─────────┘
             └────── KV Cache Transfer ───┘
                   (NIXL / RDMA)

Core advantages: Independent scaling, hardware heterogeneity (cost savings), eliminated cross-phase interference.

Layer 6: Token Billing — The Business Lifeline

Usage Data

vLLM returns usage in every response. With --enable-prompt-tokens-details, it also includes cache hit details:

{
  "usage": {
    "prompt_tokens": 3000,
    "completion_tokens": 423,
    "total_tokens": 3423,
    "prompt_tokens_details": {
      "cached_tokens": 2800
    }
  }
}

Billing Architecture

User → [API Gateway (auth/rate-limit)] → [vLLM Engine]
                  ↓                              ↓
         [Billing Middleware] ←──── Extract usage from response
                  ↓
         Billing DB (user_id, org_id, prompt_tokens, completion_tokens, timestamp)

Prometheus Metrics

# Token metering
vllm_prompt_tokens_total / vllm_generation_tokens_total

# Cache efficiency (V1, replaces deprecated gpu_prefix_cache_hit_rate)
vllm_prefix_cache_hits / vllm_prefix_cache_queries
# PromQL: (rate(vllm_prefix_cache_hits[5m]) / rate(vllm_prefix_cache_queries[5m])) * 100

# Latency
vllm_time_to_first_token_seconds / vllm_inter_token_latency_seconds

# Saturation
vllm_gpu_cache_usage_perc / vllm_num_requests_waiting / vllm_num_preemptions_total

⚠️ Must set PROMETHEUS_MULTIPROC_DIR env var for correct multi-process metric collection.

Five Token Billing Pitfalls

Pitfall 1: Should cached prefix tokens be billed?

Your actual compute cost:
  Total prompt = 3000 tokens
  APC prefix hit = 2800 tokens (cost ≈ 0)
  Actually computed = 200 tokens

User pays: Full 3000 prompt_tokens (industry convention)
Your profit = User payment for 3000 tokens - Your GPU cost for 200 tokens
Prefix caching is your profit engine.

Pitfall 2: Streaming usage only in last chunk. Extract usage from the chunk where finish_reason != null.

Pitfall 3: Speculative decoding waste tokens. vLLM's completion_tokens automatically excludes rejected draft tokens.

Pitfall 4: /tokenize endpoint mismatches. Chat Template special tokens (<|im_start|>) are included. Ensure your billing tokenizer matches vLLM's exactly.

Pitfall 5: Multimodal token counting. Images encode as hundreds-to-thousands of visual tokens reflected in prompt_tokens. Consider separate pricing.

Cost Analysis: Build vs Buy

Qwen2.5-72B (FP8) on 2x H100:
  GPU cost: ~$7/hour = $168/day
  Throughput: ~4,000 tokens/s = ~345M tokens/day

vs. OpenAI GPT-4o ($2.50/M input + $10/M output):
  Same volume: ~$1,500/day

Self-hosted: $168/day vs API: $1,500/day → ~9x cheaper

Rule of thumb:
  < 10M tokens/day → Buy API (simpler)
  10M-100M tokens/day → Evaluate based on team capability
  > 100M tokens/day → Self-host (cost advantage is overwhelming)

Production Checklist

I. GPU & Compute

Item	Recommended	Notes
GPU	H100 SXM 80GB / A100 80GB	H100 supports FP8 zero-calibration
Interconnect	NVLink 4.0 / NVSwitch	TP GPUs must be NVLink (not PCIe)
CPU	≥ (2 + N) physical cores (N=GPU count)	Engine Core is CPU-sensitive
RAM	≥ model file size × 2	For model loading

II. Storage & Model Loading

Item	Recommended	Notes
Model storage	Local NVMe SSD / fast NFS	Cold start from HuggingFace is too slow
K8s	PersistentVolume + InitContainer	Pre-download via Job

III. Network & Security

Item	Recommended	Notes
Multi-node	InfiniBand RDMA / RoCE	Required for disaggregated inference
External	Reverse proxy (Nginx/Kong)	vLLM’s `--api-key` isn’t production security
Isolation	Private VPC	vLLM inter-node comms are unencrypted by default

IV. vLLM Engine Params

vllm serve /models/Qwen2.5-72B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \                    # ⚠️ MOST CRITICAL
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \          # Expose cached_tokens
  --max-num-seqs 256 \
  --quantization fp8 \
  --kv-cache-dtype fp8_e5m2 \
  --tensor-parallel-size 2

V. Monitoring

Component	Recommended	Key Panels
Metrics	Prometheus	Cache hit rate, KV Cache %, queue depth
Dashboards	Grafana	6 panels: cache hits, throughput, latency, queue, preemptions, TTFT P99
Logging	ELK / Loki	Request-level traces
Tracing	OpenTelemetry	End-to-end latency

VI. Billing & Business

Component	Recommended	Notes
Gateway	Kong / Nginx / Traefik	Auth, rate limiting, routing
Billing middleware	Custom Python/Go	Extract `usage` + `cached_tokens`
Database	TimescaleDB / ClickHouse	Time-series billing records

VII. High Availability

Item	Recommended	Notes
Load balancer	Nginx / Traefik / K8s Service	Multi-replica traffic
Health checks	Active inference probes	Don’t just check process liveness
Orchestration	Kubernetes + KEDA	Autoscale on queue depth
Rolling updates	Start new before stopping old	Model loading takes 30s+

Summary

Key action items:

Get running first, tune later: Start with a simple vllm serve, validate business logic, then optimize.
max-model-len is priority #1: This single parameter may outweigh all other optimizations combined.
Monitor first: Set up Prometheus + Grafana. gpu_cache_usage_perc is the single most important metric.
Cache hit rate = profit margin: Track APC hits, optimize prompt structure to front-load static content.
Bill at the gateway: Don't modify vLLM source code. Intercept usage at the gateway layer.

Stay updated via official docs and GitHub Releases.

Frequently Asked Questions (FAQ)

Q1: How to troubleshoot CUDA out of memory (OOM) when launching vLLM?

First inspect --gpu-memory-utilization (default 0.90 allocates 90% of GPU VRAM for weights and KV Cache); adjust it to 0.85 if other background processes consume VRAM. Most importantly, explicitly configure --max-model-len (e.g. 4096 or 8192) according to your practical workload instead of letting it allocate for the theoretical context window. If VRAM remains constrained, enable 4-bit/8-bit weight quantization by consulting our Quantization Hands-on Guide.

Q2: When should I choose Tensor Parallelism vs Pipeline Parallelism?

On a single machine with multiple GPUs connected via NVLink (such as 4x or 8x A100/H100), Tensor Parallelism (--tensor-parallel-size) is preferred because NVLink communication latency is negligible. For multi-node distributed setups, the best practice is multi-instance Data Parallelism with a lightweight load balancer, which prevents inter-node network synchronization bottlenecks.

Q3: What makes vLLM superior to Ollama or TGI for production serving?

vLLM is specifically engineered for high-concurrency enterprise throughput. Utilizing PagedAttention, Continuous Batching, and Chunked Prefill with Automatic Prefix Caching (APC), vLLM achieves 3x to 5x higher sustained throughput under variable batch requests while exposing a native OpenAI-compatible API. For hardware background, refer to our NVIDIA GPU Package Architecture breakdown.

Originally published at Nobita Talks AI on blog.llmgo.top.

Speculative Decoding in Production: From Draft Models to EAGLE-3 Dynamic Trees for 3x-5x Lossless Acceleration

ifnodoraemon — Sun, 20 Sep 2026 04:51:57 +0000

Introduction: The Memory-Bandwidth Curse of Autoregressive Decoding

Before evaluating model acceleration techniques, we must confront the primary physical bottleneck of LLM inference: autoregressive generation is profoundly memory-bandwidth bound.

Consider serving an unquantized 70B parameter model in FP16 at single concurrency (Batch Size = 1):

The static model weights occupy 140 GB of VRAM;
To generate a single new token, the GPU must fetch all 140 GB of weights from High Bandwidth Memory (HBM) into on-chip SRAM and registers;
On a premier GPU delivering 3 TB/s of memory bandwidth, moving 140 GB requires approximately 46 milliseconds;
During those 46 milliseconds, the GPU Tensor Cores perform minimal floating-point operations. For over 95% of each decoding cycle, expensive compute units idle waiting on memory bus transfers.

graph LR
    subgraph Traditional Autoregressive Decoding: Sequential Memory Stalls
        A1["Fetch 140GB Weights"] --> A2["Generate Token 1 (46ms)"]
        A2 --> A3["Fetch 140GB Weights"] --> A4["Generate Token 2 (46ms)"]
        A4 --> A5["Fetch 140GB Weights"] --> A6["Generate Token 3 (46ms)"]
    end
    subgraph Speculative Decoding: Speculative Drafting & Parallel Verification
        B1["Ultra-lightweight Draft Unit (5ms)"] --> B2["Speculatively generate 5 candidate tokens"]
        B2 --> B3["Single target forward pass (Fetch 140GB once, 48ms)"]
        B3 --> B4["Accept 4~5 tokens concurrently! (Throughput multiplied)"]
    end

To eliminate this memory bandwidth bottleneck, Speculative Decoding shifts the workload from memory-bound sequential fetching to compute-bound parallel verification, generating multiple tokens in a single forward pass of the target model.

I. Mathematical Foundation: The Proof of Exact Lossless Invariance

Engineers often ask: "Does guessing tokens with a smaller auxiliary model degrade output quality or drift the target probability distribution?"

The answer is mathematically definitive: The output distribution of speculative decoding is provably and strictly 100% identical to the target base model.

1. Verification via Modified Rejection Sampling

Let the target LLM be $M_p$ with conditional distribution $p(x)$, and the speculative draft mechanism be $M_q$ with distribution $q(x)$.

Suppose the draft mechanism speculatively generates $K$ consecutive candidate tokens: $(x_1, x_2, \dots, x_K)$. The target model $M_p$ executes a single parallel forward pass across all $K$ positions, evaluating the target probabilities: $p(x_1), p(x_2), \dots, p(x_K)$.

For candidate token $x_k$, the engine executes a modified rejection sampling check:

Calculate Acceptance Probability: $$\alpha = \min\left(1, \frac{p(x_k)}{q(x_k)}\right)$$
Sample Uniform Random Variable $r \sim \text{Uniform}(0, 1)$:
- If $r \le \alpha$, accept token $x_k$;
- If $r > \alpha$, reject token $x_k$ and terminate verification for subsequent tokens in this draft branch.
Residual Resampling upon Rejection: If a token is rejected at position $k$, the engine samples an alternate replacement token directly from the normalized positive residual distribution: $$P_{resample}(x) = \frac{\max(0, p(x) - q(x))}{\sum_x \max(0, p(x) - q(x))}$$

The Invariance Theorem: As rigorously proven by Leviathan et al. (2023), marginalizing the joint distribution over the acceptance and residual branches recovers the exact target distribution $p(x)$. Whether employing greedy decoding or temperature-based stochastic sampling, output fidelity is mathematically preserved.

II. The Four Generations of Speculative Decoding

Speculative decoding has evolved through four major architectural generations:

Generation	Architectural Principle	Primary Strengths	Bottlenecks & Limitations	Key Citations
Gen 1: Dual-Model Drafting (Draft-Target)	A small dense model (e.g. Llama-3.2-1B) generates drafts for a 70B target model.	Conceptually simple; uses off-the-shelf pre-trained models.	The draft model is still a full transformer, competing for HBM memory bandwidth on busy GPUs.	Leviathan et al. (2023)
Gen 2: Parallel Multi-Head Prediction (Medusa)	Eliminates independent draft models; appends multiple parallel MLP prediction heads directly to the target model's final transformer layer.	Zero additional model loading; no independent base model.	Heads lack causal self-attention across draft positions, causing acceptance rates to collapse beyond 3 tokens.	Medusa (2024)
Gen 3: Feature Extrapolation & Dynamic Trees (EAGLE-1 / EAGLE-2)	Autoregressively extrapolates features in the penultimate hidden state and introduces Context-Aware Dynamic Draft Trees.	Smooth feature representations boost acceptance rates above 80%, yielding >3x speedup.	Requires training a lightweight autoregressive head per target architecture.	SafeAILab / Tsinghua (2024)
Gen 4: Multi-Scale Semantic Fusion (EAGLE-3)	Fuses low-, mid-, and high-level hidden representations across transformer depths into the dynamic tree.	Significantly improves confidence on rare tokens and code syntax, unlocking 4x~5.6x speedups.	Requires distributed offline synthetic feature extraction pipelines during training.	EAGLE-3 (2025/2026)

III. Dissecting EAGLE: Why Dynamic Draft Trees Dominate

Tsinghua University's open-source framework EAGLE (SafeAILab/EAGLE) has become the standard speculative backend across both vLLM and SGLang.

1. From Linear Chains to Dynamic Trees

Traditional speculative drafting predicts a single linear sequence: $Token_1 \to Token_2 \to Token_3$.

The Linear Bottleneck: If the model has 95% confidence on Token 1, but Token 2 represents an ambiguous conjunction with only 40% confidence, a rejection at Token 2 invalidates all subsequent tokens, even if Tokens 3 and 4 were entirely accurate!

graph TD
    subgraph Traditional Linear Drafting (Cascade Failure)
        L1["Token A (95% Confidence - Match)"] --> L2["Token B (40% - Rejected!)"]
        L2 -.->|Downstream Discarded| L3["Token C (Accurate)"]
        L2 -.->|Downstream Discarded| L4["Token D (Accurate)"]
    end
    subgraph EAGLE-2/3: Context-Aware Dynamic Draft Tree (Tree Attention)
        T0["Root Context"] --> T1["Candidate Token A (95%)"]
        T1 --> T2["Branch B1 (45%)"]
        T1 --> T3["Branch B2 (40%)"]
        T2 --> T4["Branch C1 (90%)"]
        T3 --> T5["Branch C2 (85%)"]
    end

2. The Tree Attention Verification Mechanism

EAGLE flattens the multi-branch candidate tree into a single concatenated sequence, using a 2D Tree-Attention Mask to allow the target model to verify all candidate branches in one single forward pass:

If branch $B_1$ is rejected but branch $B_2$ matches, the engine accepts path $B_2 \to C_2$ seamlessly.
Dynamically pruning unlikely paths keeps the average accepted tokens per step ($\tau$) reliably between 3.5 and 4.8.

IV. Production Deployment: vLLM & SGLang Integration

Here is how to deploy EAGLE speculative decoding for Qwen/Qwen2.5-72B-Instruct in production containers.

1. Deploying with vLLM

vLLM natively integrates EAGLE via the --speculative-model parameter:

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --speculative-model yuhuili/EAGLE-Qwen2.5-72B-Instruct \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1 \
  --port 8000

Key Flag Rationale:

--speculative-model: Points to the dedicated EAGLE lightweight head weights on HuggingFace (~500MB to 1GB);
--num-speculative-tokens 5: Drafting 5 tokens balances verification throughput with GPU latency;
--speculative-draft-tensor-parallel-size 1: The draft head is small enough to run on a single GPU without cross-GPU communication overhead.

2. Deploying with SGLang

SGLang provides native tree-search kernels for EAGLE (for architectural comparisons, see our SGLang vs vLLM Architecture Breakdown):

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-72B-Instruct \
  --speculative-algorithm EAGLE \
  --speculative-draft yuhuili/EAGLE-Qwen2.5-72B-Instruct \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 4 \
  --speculative-num-draft-tokens 16 \
  --tp 4 \
  --port 30000

Note: --speculative-eagle-topk 4 and --speculative-num-draft-tokens 16 activate dynamic tree exploration with depth 5 across 16 draft nodes.

V. When Does Speculative Decoding Fail? The Slowdown Trap

Speculative decoding is not universally beneficial. Under certain operating conditions, it can cause an inadvertent performance regression (Negative Speedup):

graph TD
    Start["Analyze Serving Workload"] --> Q1{"What is the target concurrency level?"}
    Q1 -->|"High Concurrency (Tensor Cores Already 100% Saturated)"| Slowdown["⚠️ Disable Speculative Decoding<br/>(Verification passes steal FLOPs from normal batches, reducing total throughput)"]
    Q1 -->|"Low-to-Medium Concurrency (Latencies are Memory-Bound)"| Q2{"What is the entropy/predictability of the generation task?"}

    Q2 -->|"High-Entropy Content (Open-ended creative writing / random keys)"| Fragile["⚠️ Acceptance rate drops below 30%<br/>(Draft overhead exceeds verification gains)"]
    Q2 -->|"Structured & Deterministic (Code, JSON, Math, Translation)"| SuperFast["🚀 Enable EAGLE<br/>(Acceptance rate >80%, 3.5x~5x acceleration)"]

The Compute Saturation Boundary

Low Concurrency (Concurrency $\le 16$): GPU Tensor Cores sit underutilized during decoding. Speculative verification leverages idle compute to accelerate Inter-Token Latency (ITL) by 60%~75%.
High-Throughput Concurrency (Concurrency $\ge 128$): Batches are already dense enough to saturate memory bandwidth into a Compute-Bound state. Adding speculative verification passes can reduce overall system throughput by 10%~15%.

Frequently Asked Questions (FAQ)

Q1: Does speculative decoding require retraining the base model?

No. The base foundation model remains 100% frozen. EAGLE trains an auxiliary single-layer decoder head on frozen intermediate feature vectors, requiring only 0.5% to 1% of the base model's parameters and a few hours of commodity GPU training.

Q2: Can speculative decoding be combined with FP8 weight quantization?

Yes, and this represents the standard high-performance inference stack in 2026. Deploying FP8/AWQ weights fits models within tight VRAM constraints (see our Practical Quantization Guide), while an EAGLE speculative head bypasses memory-bandwidth bottlenecks.

Q3: Why does real-world speedup sometimes lag behind theoretical benchmarks?

Realized acceleration is strictly bounded by the empirical acceptance rate $\alpha$. In programming languages (Python/Java) and JSON extraction where syntax is highly structured, acceptance rates exceed 85%, yielding 4x+ speedups. In open-ended conversational domains with high sampling temperatures ($T \ge 1.0$), acceptance rates drop toward 50%, yielding more modest 1.8x~2.2x speedups.

Originally published at Nobita Talks AI on blog.llmgo.top.

SGLang vs vLLM Architecture Showdown: RadixAttention, Structured Decoding, and High-Concurrency Benchmarks

ifnodoraemon — Sun, 20 Sep 2026 04:51:51 +0000

Introduction: The Dual Titans of Open-Source Inference

Throughout 2024 and early 2025, vLLM, developed by UC Berkeley's Sky Computing Lab, established itself as the undisputed de facto standard for open-source LLM inference serving, largely due to its pioneering PagedAttention algorithm.

However, as 2026 arrived, the paradigm shifted toward agentic workflows, multi-turn tool-calling chains, and strict JSON Schema constraints. In this landscape, another framework from Berkeley's LMSYS team—SGLang—rose rapidly to challenge vLLM's dominance.

Engineering teams across the industry now face a critical architecture dilemma:

Given that vLLM supports Automatic Prefix Caching (APC) and continuous batching, why migrate to SGLang?
How does SGLang's RadixAttention physically outperform traditional block-level PagedAttention?
What is the true Time-To-First-Token (TTFT) and throughput delta in multi-turn Agent loops and structured JSON extraction?

This article moves past marketing claims to analyze the underlying memory data structures and scheduler-level compilation pipelines, supported by empirical benchmarks on an 8x NVIDIA H100 SXM cluster.

I. Core Memory Architecture: PagedAttention vs. RadixAttention

Because autoregressive generation is fundamentally memory-bandwidth bound, the core differentiator of any inference engine is how efficiently it allocates, retains, and reuses the KV Cache.

graph TD
    subgraph vLLM: PagedAttention (Block Paging & Flat Matching)
        V1["Incoming Prompt Tokens"] --> V2["Slice into Fixed Blocks (e.g. 16 Tokens)"]
        V2 --> V3["Hash Map Lookup for Matching Block Sequences"]
        V3 --> V4["Discrete Memory Mapping (Similar to OS Virtual Memory)"]
    end
    subgraph SGLang: RadixAttention (Dynamic Tree-Structured Caching)
        S1["Incoming Prompt Tokens"] --> S2["Traverse Radix Tree Top-Down"]
        S2 --> S3{"Shared Sub-tree Match?"}
        S3 -->|"Full Match"| S4["Zero-Overhead Pointer Reuse to KV Pages"]
        S3 -->|"Partial Match"| S5["Split Node at Exact Divergence Point"]
        S4 --> S6["LRU Tree Pruning Strategy"]
        S5 --> S6
    end

1. vLLM's PagedAttention and Flat Hash Caching

vLLM's breakthrough was adapting virtual memory paging principles from operating systems to GPU tensors. By dividing dynamic KV caches into fixed-size "blocks" (typically 16 or 32 tokens) linked via page tables, it eliminated memory fragmentation.

To achieve prompt reuse, vLLM introduced Automatic Prefix Caching (APC) based on a hash-chained cache pool:

When a request arrives, the engine computes cryptographic hashes across sequential token blocks.
It checks the cache pool for matching blocks and chains them together.
The Engineering Limitation: vLLM organizes cache blocks linearly and flatly. In branching workflows (such as Monte Carlo Tree Search, Agent rollbacks, or prompts with interspersed variables), flat hashing struggles to capture dynamic tree forks. Blocks often miss the cache or are prematurely evicted.

2. SGLang's RadixAttention: The Tree-Structured Paradigm

SGLang fundamentally reimagines memory caching by introducing RadixAttention.

Instead of treating the KV cache as disjointed linear blocks, it organizes all active and retained KV caches into a global Radix Tree (Compressed Prefix Trie):

Paths Represent Prefixes: The root node represents an empty prompt. Every directed edge carries a sequence of tokens, while internal nodes and leaves point to physical GPU memory pages holding the corresponding KV tensors.
Adaptive Node Splitting: If two requests share a lengthy System Prompt but diverge on an intermediate tool output, the Radix Tree splits the node at the exact token divergence point. Both requests share the parent node's KV cache with zero redundancy, allocating memory only for the differential branch.
Topological LRU Eviction: When GPU VRAM approaches capacity, SGLang prunes the least recently used leaf nodes, allowing heavily shared system prompts and root nodes to remain resident indefinitely.

Benchmark Difference: For one-off stateless prompts, both engines perform similarly. But in multi-turn chat, iterative tree searches, and Agent workflows with shared system prompts, SGLang's cache hit rate surges from vLLM's 30%~45% up to 70%~85%, cutting TTFT by up to 4x. For the underlying mathematical foundations of memory compression, see our analysis on Kimi KDA and DeepSeek MLA Architecture.

II. Structured Decoding: External Logit Masking vs. Scheduler-Level FSM Compilation

In production environments, over 60% of API endpoints require strict compliance with JSON Schemas, regex patterns, or domain-specific languages (DSLs). The two frameworks handle this via completely different pipelines.

Evaluation Vector	vLLM (via Outlines / Guided Decoding)	SGLang (Scheduler-Native Compressed FSM)
Core Mechanism	External Logit Masking: At each forward step, an external regex state machine computes valid tokens and masks invalid logits to $-\infty$ prior to Softmax.	Compressed Finite State Machine (FSM): Compiles the schema directly into the C++/CUDA scheduler core, detecting deterministic tokens to trigger immediate jump-forward bypass.
Execution Layer	Interceptor wrapper between the API server and scheduler	Deeply compiled inside the inner CUDA forward loop
Literal Generation	For static syntax like `{"status": "success", "data": [`, the model must still execute full autoregressive matrix multiplications token-by-token.	Jump-Forward Decoding: Identifies deterministic strings and injects them in a single step, skipping autoregressive forward passes completely!
Throughput Under Load	At 64+ concurrent requests, CPU overhead from evaluating massive token masks (e.g. 128k vocabulary) bottlenecks the system, dropping throughput by >40%.	FSM transitions consume <5 microseconds on CPU. Throughput under strict JSON constraints remains within 95% of unconstrained generation.

sequenceDiagram
    participant Client as Application Client
    participant Scheduler as SGLang Scheduler
    participant GPU as GPU Tensor Cores

    Client->>Scheduler: Request (Strict JSON Schema Enforced)
    Note over Scheduler: Compiles JSON Schema into Optimized FSM
    Scheduler->>GPU: Generate dynamic key '{"order_id": "'
    Note over Scheduler: FSM detects subsequent quotes and commas as deterministic
    Scheduler-->>GPU: Execute Jump-forward Injection (Bypasses Matrix Multiplication)
    GPU->>Scheduler: Return dynamic token sequence
    Scheduler->>Client: Stream valid JSON (Up to 2.5x throughput gain)

III. 8x H100 Production Cluster Benchmarks

To provide concrete empirical data, we evaluated both engines on an enterprise node equipped with 8x NVIDIA H100 SXM5 80GB GPUs.

1. Benchmark Environment

Compute: 8x H100 SXM 80GB (NVLink 4.0, 900 GB/s bidirectional interconnect bandwidth)
Host: Dual Intel Xeon Platinum 8480+ (112 cores), 1TB DDR5 RAM
Model: Qwen/Qwen2.5-72B-Instruct (FP8 quantization)
Concurrency Sweep: Concurrency levels $C \in [1, 16, 64, 128, 256]$

2. Scenario A: Stateless General QA (Prompt: 2048 Tokens, Output: 512 Tokens, 0% Prefix Overlap)

Measures raw operator kernel efficiency and continuous batching throughput without caching advantages.

Concurrency ($C$)	vLLM Throughput (Tokens/s)	SGLang Throughput (Tokens/s)	vLLM P99 TTFT (ms)	SGLang P99 TTFT (ms)
1	48.2	49.1	82	80
16	690.4	702.1	145	140
64	2,410.8	2,480.3	420	410
128	4,120.5	4,190.2	890	860

Verdict: In stateless, non-overlapping workloads, performance is essentially identical. SGLang maintains a negligible 1%~3% advantage due to FlashInfer kernel tuning, while vLLM demonstrates solid, predictable behavior. For advanced vLLM tuning, consult our vLLM Production Serving Guide.

3. Scenario B: Multi-Turn Agent Tool Calling (Prompt: 4096 Tokens, 75% Prefix Overlap)

Simulates an enterprise agent conversational loop with rich tool declarations and shared conversation history.

Benchmark Results (Concurrency = 64, Prefix Overlap = 75%):
------------------------------------------------------------
Metric                         vLLM (APC Active)   SGLang (RadixTree)   Advantage
Median TTFT (P50)                   380 ms              85 ms           SGLang 4.47x Faster 🚀
Tail Latency TTFT (P99)           1,250 ms             280 ms           SGLang 4.46x Faster 🚀
KV Cache Hit Rate                    41.2%               78.6%          Almost 2x Hit Rate
Total Output Throughput (Tokens/s)   3,120               5,430          SGLang +74% Gain
------------------------------------------------------------

Architectural Analysis:
In multi-turn execution, vLLM's APC frequently misses cache hits because user variations disrupt flat hash alignment across non-contiguous blocks. SGLang's RadixTree locks the parent system prompt and historical turns into shared branches, executing prefill almost instantaneously.

4. Scenario C: Strict JSON Schema Extraction

Forces the model to parse complex unstructured financial filings into an exact 20-field nested JSON schema.

Throughput vs. Concurrency Under Structured Constraints:
- Unconstrained Generation: Both engines reach ~4,200 tokens/s at Concurrency 128
- Enforcing Strict JSON Schema:
  * vLLM (Guided Decoding): Throughput drops to 2,350 tokens/s (CPU logit masking saturation)
  * SGLang (FSM Jump-forward): Throughput sustains 3,980 tokens/s (Less than 6% degradation)

IV. Production Deployment Recipes

1. vLLM Production Configuration (Recommended for Broadest Compatibility)

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 256 \
  --quantization fp8 \
  --port 8000

2. SGLang Production Configuration (Recommended for Agent & Structured Workloads)

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-72B-Instruct \
  --tp 8 \
  --mem-fraction-static 0.90 \
  --context-length 16384 \
  --enable-flashinfer \
  --schedule-policy lpm \
  --port 30000

Note: --schedule-policy lpm activates Longest Prefix Match scheduling, maximizing radix tree hit rates.

V. Enterprise Architecture Decision Framework

graph TD
    Start["Evaluate Workload Requirements"] --> Q1{"Do queries involve multi-turn agent loops, heavy prefix sharing, or strict JSON output?"}
    Q1 -->|"Yes (Agentic / RAG / JSON-Heavy)"| BranchSGLang["Select SGLang"]
    Q1 -->|"No (Stateless QA / Batch Processing)"| Q2{"Are you deploying on non-standard accelerators or obscure model architectures?"}

    Q2 -->|"Yes (Custom NPUs / Exotic Topologies)"| BranchVLLM["Select vLLM (Widest Ecosystem Support)"]
    Q2 -->|"No (Standard NVIDIA/AMD GPUs, Mainstream LLMs)"| Q3{"Does your ops pipeline require turn-key Helm charts and commercial K8s operators?"}

    Q3 -->|"Yes (Prioritize Turn-Key Stability)"| BranchVLLM
    Q3 -->|"No (Prioritize Latency & Throughput)"| BranchSGLang

Frequently Asked Questions (FAQ)

Q1: Does RadixAttention introduce CPU overhead from frequent tree splits?

No. Radix tree operations (lookups, splits, inserts, and pointer swaps) are executed via optimized C++ and Rust structures in host memory. In typical workloads of 100 to 200 concurrent requests, tree operations execute in microseconds ($\mu s$), rendering CPU overhead negligible compared to multi-millisecond GPU matrix multiplications.

Q2: Can vLLM simply adopt RadixAttention in a future release?

Not without a complete architectural rewrite. vLLM's scheduler and distributed memory managers are deeply coupled to the PagedBlock abstraction, which coordinates cross-GPU synchronization across tensor and pipeline parallelism ranks. Transitioning vLLM to a dynamic directed acyclic graph (DAG) prefix tree would require redesigning its core scheduler. Both frameworks will maintain distinct architectures for the foreseeable future.

Q3: Does SGLang support model quantization and speculative decoding?

Yes. SGLang natively supports FP8, AWQ, GPTQ, and Marlin kernels (for quantization trade-offs, review our Practical Quantization Guide and Quantization Precision Guide). Furthermore, SGLang natively integrates dynamic tree-based speculative decoding via EAGLE.

Originally published at Nobita Talks AI on blog.llmgo.top.

Test-Time Compute and GRPO in Practice: From PPO to Critic-Free Reinforcement Learning

ifnodoraemon — Sun, 20 Sep 2026 03:13:28 +0000

Introduction: The Pre-training Wall and the Dawn of Test-Time Scaling

For the past several years, the foundational law of frontier LLM development was Chinchilla's Pre-training Scaling Laws: stack deeper transformer layers, ingest multi-trillion token corpora, and burn increasingly massive GPU clusters.

However, entering 2026, this brute-force approach has encountered formidable physical and thermodynamic bottlenecks:

The Human Knowledge Depletion Wall: High signal-to-noise public internet text has been virtually exhausted. Naive ingestion of low-quality synthetic web dumps risks "model collapse" and entropy degeneration during unsupervised pre-training.
Diminishing Returns on Model Scale: Pushing model parameters from 70B to 700B incurs exponential spikes in capital expenditures, cluster interconnect overhead, and power consumption, yet returns only marginal improvements on everyday reasoning.

As pre-training scaling slows, frontier reasoning engines (such as OpenAI o1/o3 and DeepSeek-R1) have ignited a secondary growth curve: Test-Time Compute Scaling Laws.

graph LR
    subgraph Traditional Paradigm: One-Shot Pre-training Inference
        A1["Complex Math/Coding Prompt"] --> A2["70B~400B Dense Base LLM"] --> A3["Greedy Decoding (Prone to Hallucinations)"]
    end
    subgraph Reasoning Paradigm: Test-Time Compute Scaling
        B1["Complex Math/Coding Prompt"] --> B2["Compact Base Model"] --> B3["Extended Chain-of-Thought (CoT)"] --> B4["Self-Verification & Backtracking"] --> B5["Deterministic Accurate Solution"]
    end

Rather than spending millions of dollars during pre-training to memorize answers to every conceivable question, test-time scaling trains models to allocate dynamic computation at inference time—thinking, calculating, and self-correcting before providing a response.

I. The Three Regimes of Test-Time Compute Scaling

In modern literature, extending test-time compute falls into three primary architectural regimes:

Scaling Regime	Core Mechanism	Primary Compute Bottleneck	Representative Work	Bottlenecks & Failure Modes
1. Sequential CoT Expansion	The model outputs multi-thousand token chains of thought (`<think> ... </think>`), enabling backtracking and scratchpad verification.	Autoregressive decoding latency	DeepSeek-R1, OpenAI o1	Prone to "overthinking" loops on trivial prompts; latency increases substantially.
2. Leaf-level Sampling & Voting	Parallel sampling of $N$ diverse paths, combined with majority voting or verifiers.	Batch concurrency capacity	Best-of-N, Self-Consistency	Search space is unguided; incorrect trajectories waste full GPU decode cycles.
3. Prefix-level Search with PRMs	Process Reward Models (PRMs) score intermediate steps within tree search (Beam Search / MCTS).	Step-level verifier evaluation	AlphaGo-style MCTS, Step-PRMs	Step-level PRM annotations are costly; imperfect verifiers invite "reward hacking."

The breakthrough of DeepSeek-R1 lies in fusing sequential chain-of-thought expansion with critic-free reinforcement learning, proving that pure rule-based RL can induce deep reasoning behaviors without manually engineered step-by-step PRMs.

II. From PPO to GRPO: Why Traditional RLHF Breaks on Long Reasoning Traces

For years, PPO (Proximal Policy Optimization) was the standard algorithm for post-training alignment. However, when applied to reasoning models with 10k+ token outputs, PPO collapses under severe infrastructure and mathematical constraints.

1. The Quad-Model VRAM Explosion

A classic PPO training setup requires hosting and synchronizing four distinct neural networks:

Actor Model ($\pi_\theta$): The trainable policy generating tokens.
Critic Model ($V_\phi$): Typically matching the Actor's size, tasked with estimating scalar state values $V(s)$.
Reference Model ($\pi_{ref}$): A frozen copy computing per-token KL divergence to prevent policy drift.
Reward Model ($R_\psi$): A frozen network scoring terminal outputs.

graph TD
    subgraph Traditional PPO Architecture
        P1["Actor Model (Trainable)"]
        P2["Critic Model (Trainable - Massive VRAM)"]
        P3["Reference Model (Frozen)"]
        P4["Reward Model (Frozen)"]
    end
    subgraph GRPO Architecture
        G1["Actor Model (Trainable)"]
        G2["Ref Weights / Analytical KL Calculation"]
        G3["Deterministic Environment (Python Sandbox / Unit Tests / Matcher)"]
    end

In a 70B parameter setup, loading the Actor and Critic alongside their respective AdamW optimizer states easily demands over 600GB of VRAM. This forces teams to deploy complex tensor and pipeline parallelism merely to fit the training loop. For low-level driver and memory bus topology guidelines, consult our NVIDIA GPU Package Architecture Deep Dive.

2. Value Function Drift Across Long Horizons

When a model reasons through intricate mathematical proofs, trajectories stretch across 8,000 to 16,000 tokens. Training a Critic to accurately predict the expected discounted return at every intermediate token is mathematically fragile. Critic errors amplify gradient variance, causing loss values to explode into NaNs.

III. Mathematical Derivation of GRPO: The Critic-Free Revolution

GRPO (Group Relative Policy Optimization) was pioneered by DeepSeek in the DeepSeekMath paper and scaled in DeepSeek-R1.

Its core thesis is remarkably elegant: Eliminate the Critic network entirely, sample a group of completions for each prompt, and use the group's empirical distribution as the baseline.

1. Group Sampling and Normalized Advantage

For any input query $q$, the policy $\pi_{\theta_{old}}$ generates a group of $G$ distinct candidate completions:

$${o_1, o_2, \dots, o_G} \sim \pi_{\theta_{old}}(q)$$

The verification environment (e.g., a regex answer parser or a compiler test runner) assigns scalar rewards to each completion:

$${r_1, r_2, \dots, r_G}$$

Rather than evaluating an absolute value network $V(s)$, GRPO computes the relative advantage $A_i$ of completion $o_i$ normalized against its peers:

$$A_i = \frac{r_i - \text{mean}({r_1, \dots, r_G})}{\text{std}({r_1, \dots, r_G}) + \epsilon}$$

If $o_i$ outperforms the group average, $A_i > 0$, reinforcing the token trajectory.
If $o_i$ underperforms, $A_i < 0$, penalizing the trajectory.
Normalizing by the standard deviation dynamically stabilizes variance across batches.

2. The GRPO Objective Function

Retaining the clipped surrogate mechanism from PPO, GRPO optimizes the following objective:

$$\mathcal{J}{GRPO}(\theta) = \mathbb{E}{q \sim P(Q), {o_i}{i=1}^G \sim \pi{\theta_{old}}(q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min \left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})} A_{i,t}, \; \text{clip}\left(\frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})}, 1-\epsilon, 1+\epsilon\right) A_{i,t} \right) - \beta D_{KL}(\pi_\theta \parallel \pi_{ref}) \right) \right]$$

where the per-token KL divergence approximation is computed directly:

$$D_{KL} = \frac{\pi_{ref}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - \log \frac{\pi_{ref}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)} - 1$$

This architectural shift achieves two immediate advantages:

Over 50% Reduction in Training Memory: Eliminating the Critic model discards its weights, gradients, and optimizer states.
Zero Value-Model Misalignment: The baseline is derived directly from empirical sampling statistics, providing stable convergence.

IV. The "Aha Moment" and the Emergence of Self-Reflection

In the DeepSeek-R1-Zero experiment, researchers demonstrated that without a single human SFT demonstration, training a pure base model with GRPO induced complex reasoning behaviors:

1. Deterministic Rule-Based Rewards

Instead of relying on a learned neural reward model (which often suffers from reward gaming), the environment enforces strict rule-based verifiers:

Accuracy Reward: Verifies if the mathematical result matches ground truth or if the unit tests pass in a sandboxed interpreter (1.0 for success, 0.0 for failure).
Format Reward: Enforces that reasoning is encapsulated strictly between <think> and </think> tags, with the final answer inside <answer>.

2. Spontaneous Emergence of Self-Correction

After several thousand iterations, as trajectory length expanded, logs revealed models discovering self-correction unprompted:

User: Is 9.11 larger than 9.9?
Assistant:
<think>
First, compare 9.11 and 9.9.
The integer part is both 9.
Now compare the tenths place: 9.11 has 1 in the tenths place, while 9.9 has 9 in the tenths place.
Wait! Let me rethink this... Am I confusing decimal places with software version numbers?
In software engineering, v9.11 is newer than v9.9, but mathematically, 9.9 = 9.90.
Since 9.90 > 9.11, 9.9 is strictly larger.
Let me double check the calculation: 9.9 - 9.11 = 0.79 > 0. Yes, absolutely correct.
</think>
<answer>
No, 9.9 is larger than 9.11.
</answer>

From a reinforcement learning perspective, exploratory paths that verified intermediate results achieved a higher pass rate on difficult tasks than one-shot guesses. The group-relative advantage mechanism amplified these self-questioning trajectories.

V. Hands-on Implementation: A Minimal GRPO Training Pipeline

Using Hugging Face's TRL (Transformer Reinforcement Learning) library, here is an end-to-end runnable script training a lightweight base model (such as Qwen/Qwen2.5-1.5B-Instruct) with GRPO:

1. Environment Setup

pip install torch transformers trl peft datasets accelerate

2. Complete Training Code

import re
import torch
from datasets import Dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import GRPOTrainer, GRPOConfig

# 1. Prepare deterministic verification dataset
train_data = [
    {
        "prompt": "Solve this equation: 3 * x + 7 = 22. What is x? Present your reasoning inside <think> and final value in <answer>.",
        "target": "5"
    },
    {
        "prompt": "A train travels 180 km in 3 hours. What is its speed in km/h? Think first in <think>, give value in <answer>.",
        "target": "60"
    },
    {
        "prompt": "If a square has an area of 64 cm^2, what is its perimeter in cm? Reason in <think>, answer in <answer>.",
        "target": "32"
    }
] * 100  # Expand dataset scale

dataset = Dataset.from_list(train_data)

# 2. Define deterministic rule-based reward functions
def correctness_reward_func(prompts, completions, target, **kwargs):
    """Verify if the content in <answer> strictly matches ground truth."""
    rewards = []
    for completion, true_target in zip(completions, target):
        match = re.search(r"<answer>(.*?)</answer>", completion, re.DOTALL)
        if match:
            pred = match.group(1).strip()
            rewards.append(2.0 if pred == true_target.strip() else 0.0)
        else:
            rewards.append(0.0)
    return rewards

def format_reward_func(completions, **kwargs):
    """Reward proper reasoning tag encapsulation."""
    rewards = []
    pattern = r"^<think>.*?</think>\s*<answer>.*?</answer>$"
    for completion in completions:
        if re.search(pattern, completion.strip(), re.DOTALL):
            rewards.append(0.5)
        else:
            rewards.append(0.0)
    return rewards

# 3. Load model and tokenizer
model_id = "Qwen/Qwen2.5-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 4. Configure GRPO Hyperparameters
training_args = GRPOConfig(
    output_dir="./grpo_output_qwen",
    learning_rate=2e-5,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_generations=4,          # Group size G=4
    max_prompt_length=256,
    max_completion_length=1024,  # Ample space for CoT exploration
    temperature=0.7,
    warmup_ratio=0.1,
    logging_steps=10,
    max_steps=100,
    save_strategy="steps",
    save_steps=50,
    bf16=True,
    report_to="none"
)

# 5. Launch the Critic-Free Trainer
trainer = GRPOTrainer(
    model=model_id,
    reward_funcs=[correctness_reward_func, format_reward_func],
    args=training_args,
    train_dataset=dataset,
)

print("🚀 Launching critic-free GRPO reinforcement learning pipeline...")
trainer.train()

For foundational training workflows and adapter memory tuning, review our Comprehensive LLM Fine-Tuning Guide.

VI. Production Pitfalls: Overthinking and Dynamic Compute Governance

Deploying reasoning models in production requires addressing these operational considerations:

The Overthinking Trap: When asked trivial factual queries ("What is the capital of France?"), models may output 800 tokens of self-questioning, adding seconds of unnecessary Time-To-First-Token (TTFT) latency.
Two-Stage Routing Architecture:
- Direct standard conversational requests and retrieval tasks to lightweight models or RAG retrieval pipelines.
- Reserve thinking endpoints for complex logical synthesis, math, and code generation with bounded token limits (max_thinking_tokens). For high-throughput infrastructure setup, refer to our vLLM Production Serving Guide.

Frequently Asked Questions (FAQ)

Q1: How does GRPO converge reliably without an explicit Critic network?

GRPO replaces the parametric state-value estimation of the Bellman equation with empirical Monte Carlo group sampling. By generating a group of $G$ responses for the same prompt, the group mean serves as a dynamic, unbiased baseline. As long as the group size is sufficient ($G \ge 4 \sim 8$), the normalized advantage $\frac{r_i - \mu}{\sigma}$ accurately signals relative trajectory quality.

Q2: Does GRPO eliminate the need for Supervised Fine-Tuning (SFT) data entirely?

While DeepSeek-R1-Zero proved that cold-start reasoning can emerge from pure RL, practical production workflows benefit significantly from a lightweight initial SFT phase. Pure RL on raw base models frequently generates multilingual gibberish, formatting anomalies, and infinite repetition early in training. Starting with a few thousand curated chain-of-thought demonstrations accelerates convergence by over 5x while preserving readability.

Q3: What is the fundamental difference between GRPO and DPO?

DPO (Direct Preference Optimization) is an offline supervised preference algorithm operating on static pairs of chosen and rejected responses $(y_w, y_l)$. It cannot discover novel reasoning pathways absent from the static dataset. GRPO is an active, online reinforcement learning algorithm where the model generates real-time samples evaluated dynamically by verifiable environment rewards, enabling open-ended exploration and spontaneous self-correction.

Originally published at Nobita Talks AI on blog.llmgo.top.

AI Coding Mastery: From 'Build Me an X' to Architecture Orchestrator

ifnodoraemon — Sun, 20 Sep 2026 03:10:37 +0000

Preface: You Might Be "Used by AI" Instead of "Using AI"

Have you ever experienced any of these?

You ask AI to build a feature, it delivers 500 lines of code, and after 30 minutes of review you realize it used a framework your project explicitly forbids
By turn 30 of the conversation, AI starts contradicting itself, overturning decisions it made earlier
AI's code "looks correct," but crashes in production — because it didn't handle the null edge cases in your business logic
You switched from Copilot to Cursor hoping for better results, but the same pitfalls persist

This isn't AI being incompetent, nor is it about picking the wrong tool — your approach to wielding AI is wrong.

In 2026, AI coding tools have evolved from "Tab completion" to autonomous Agent systems capable of planning, executing, and verifying on their own. Claude Code can run your test suites directly; GitHub Copilot Coding Agent auto-creates PRs from Issues; Cline CLI executes commands in sandboxes. Tools are abundant — what's missing is the methodology to use them correctly.

This article answers one core question: How can you make AI reliably and consistently produce production-grade code?

Structure:

Part 1: Mindset Shift ——— From Coder to Architecture Orchestrator
Part 2: Six Core Methods — Practical Playbook
Part 3: Five Anti-Patterns — Pitfall Guide
Part 4: Tool Landscape ——— 20+ Tool Selection Matrix

Part 1: Mindset Shift — Your Role Has Changed

graph LR
    subgraph Old Model
        A1["You write code"] --> B1["AI autocompletes"]
    end
    subgraph New Model
        A2["You define intent"] --> B2["AI plans"]
        B2 --> C2["AI implements"]
        C2 --> D2["AI verifies"]
        D2 --> E2["You review"]
    end

In 2026, a developer's role has shifted from "person who writes code line by line" to "architect + coach." When Andrej Karpathy coined "Vibe Coding" in 2025, saying "I just see things, say things, run things, and copy-paste things," many misinterpreted this as "casually let AI write code." The opposite is true — experts spend 70% of their time on defining constraints, reviewing plans, and encoding lessons learned.

Your core value is no longer writing syntax, but four things:

Define Intent — Clearly describe "what to do" and "what NOT to do," giving AI a quantifiable success criterion
Design Constraints — Use rule files (AGENTS.md) and specifications to set boundaries for AI
Verify Results — Use tests, linters, and adversarial reviews to confirm AI output meets architecture standards
Encode Lessons — After every mistake, encode the fix into rule files so AI permanently learns

The behavioral gap between experts and beginners is stark:

Beginner Approach	Expert Approach
"Build me a login system"	"Don't write code yet. Read the project structure first, then give me an implementation plan"
Ask AI to do everything at once	Break into atomic tasks, verify each before continuing
Accept AI's first suggestion	Ask AI for 2-3 options, analyze trade-offs, then choose
Merge code they don't understand	"Explain why this code is written this way"
Re-prompt when AI makes mistakes	Encode the error into project rule files so AI never repeats it
Use one endless chat for everything	Summarize after each sub-task, start fresh sessions

Part 2: Six Core Methods

Method 1: Spec-Driven Development

Core idea: Make AI understand "what" before it writes code. Changing a plan is always 10x cheaper than changing code.

A complete Spec-Driven cycle has four phases:

Phase 1: SPECIFY
│  "What to build, what NOT to build, what defines success"
│  → Output: SPEC.md
│
Phase 2: PLAN
│  AI proposes architecture in read-only mode; you review and refine
│  → Output: PLAN.md
│
Phase 3: TASKS
│  Decompose the plan into independently verifiable atomic tasks
│  → Output: TASKS.md
│
Phase 4: IMPLEMENT + VERIFY
   Execute tasks one by one → test → proceed only after passing
   → Output: Working code + tests

Practical tips:

Not every task needs a Spec. Rule of thumb: if it's cross-session, high-risk, or multi-file, write a Spec; if it's a single function and low-risk, just do it
Use Plan Mode: In Claude Code, say "Don't write code yet, give me a plan"; in Cursor, describe before starting the Agent
Keep documents alive: Have AI update SPEC.md in real-time during implementation to keep plan and reality in sync

Rule of thumb: If you find yourself repeatedly asking AI to "redo," you skipped the SPECIFY and PLAN phases. Go back to the beginning and align on intent.

Method 2: Context Engineering

Core idea: "Prompt Engineering" is outdated. The 2026 key skill is "Context Engineering" — not how to ask questions, but how to make the right information automatically appear at the right time.

AI's output quality is a function of context quality: Output = f(Context). What you feed it is what you get back.

The Three-Layer Context Architecture

graph TB
    subgraph Three-Layer Context
        L1["Always-On Layer"] --> |AGENTS.md / .cursorrules| Note1["Minimal base rules, auto-loaded every time"]
        L2["Auto-Attached Layer"] --> |.cursor/rules/*.mdc| Note2["Activated by file path"]
        L3["Session Layer"] --> |"@ references in chat"| Note3["Specific files for current task"]
    end

Project Rule Files — The "Job Description" for AI

This is the most critical infrastructure for mastering AI. Without it, even the best AI can only give you generic code that may violate project conventions.

File	Purpose	Tools	Priority
`AGENTS.md`	Universal "machine README"	All platforms (Cursor / Copilot / Claude Code...)	⭐⭐⭐ Must create
`CLAUDE.md`	Claude-specific instructions	Claude Code	⭐⭐ Recommended for Claude users
`.cursor/rules/*.mdc`	Layered rule system	Cursor	⭐⭐ Recommended for Cursor users
`.github/copilot-instructions.md`	Copilot global instructions	GitHub Copilot	⭐
`.goosehints`	Goose instructions	Goose	⭐

What does a good AGENTS.md look like? Here's an example from my Hugo tech blog:

# Project: ifnodoraemon.github.io (Hugo Tech Blog)

## Tech Stack
- Hugo SSG + Vanilla JS + CSS
- Bilingual architecture: content/zh/ and content/en/

## Key Commands
- hugo server -D     # Local preview
- npm run build      # Production build

## Article Standards
- Chinese articles go in content/zh/articles/{slug}.md
- English articles go in content/en/articles/{slug}.en.md
- Frontmatter must include: title / slug / date / tag / tagClass / description
- tagClass options: tag-blue / tag-green / tag-violet / tag-emerald

## Writing Style
- Open with pain-point scenarios, never write "This article will introduce..."
- Deep technical analysis + directly copyable code
- Use mermaid diagrams and comparison tables

## Safety Boundaries
- ❌ Never modify the public/ directory (build output)
- ❌ Don't introduce new CSS frameworks
- ❌ Don't modify .github/workflows/ CI configs

Note: This file is only 25 lines. Brevity is key — every useless instruction dilutes the truly important rules.

Four Golden Rules:

Keep it under 200 lines — LLMs suffer from the "lost-in-the-middle" effect: attention to information in the middle of context is lowest. Overly long rules get ignored
Only write what AI can't infer — Don't repeat what linters and type checkers already enforce. "Use TypeScript" is unnecessary; "API routes use kebab-case" is essential
Iterate through friction — AI keeps making the same mistake (e.g., always forgetting tagClass in Hugo frontmatter)? Immediately encode it into the rule file
Provide benchmark file paths — Instead of lengthy descriptions, "New articles should follow the format in content/zh/articles/mcp-guide.md" says it all

Cross-tool universality is AGENTS.md's killer advantage. Whether you use Cursor, Copilot, Claude Code, or Aider, they all automatically read this file from the project root. Configure once, effective everywhere. For modular domain capabilities and workflow automation, explore our deep dive on Skills for AI Coding Assistants.

Method 3: TDD + Agent Verification Loop

Core idea: Use tests as AI's "brake system." Let tests tell AI if it's right, instead of relying on your eyeball review.

This is the single most reliable AI coding pattern. Period. The reason is simple — AI excels at "given a quantifiable target, iterate until convergence."

graph TD
    A["You write tests: define 'what correct means'"] --> B["AI writes implementation"]
    B --> C{"Run tests"}
    C -->|Pass| D["✅ Next task"]
    C -->|Fail| E["Error logs fed back to AI"]
    E --> F["AI self-corrects"]
    F --> C

Why is this the most reliable pattern?

Quantifiable success criteria: Not "does it look good" (subjective), but "do tests pass" (objective fact)
Automatic guardrails: When AI modifies code later, existing tests immediately catch regressions
Tests are documentation: Tests are the best behavioral documentation

Practical workflow:

You write red tests (define expected behavior)
AI writes green implementation (make tests pass)
AI refactors (tests stay green)

Real scenario: Suppose you want to add an RSS generator to your blog. Don't say "build me an RSS feature" — write tests first:

# test_rss.py — You write this
def test_rss_contains_latest_articles():
    feed = generate_rss(articles[:10])
    assert "<rss version" in feed
    assert articles[0].title in feed

def test_rss_escapes_html_in_description():
    article = Article(title="Test", description="<script>alert('xss')</script>")
    feed = generate_rss([article])
    assert "<script>" not in feed  # Must be escaped

Then tell AI: "Implement the generate_rss function to make all tests pass." AI receives an executable specification, not a vague natural language description.

In terminal Agents like Claude Code / Codex CLI / Aider, AI can directly run pytest and read error output, automatically entering the Red→Green→Refactor cycle until all tests pass.

Method 4: Multi-Agent Orchestration (CIV Pattern)

Core idea: Don't let one AI simultaneously be "architect," "coder," and "tester." Separate roles, separate responsibilities.

The CIV (Coordinator-Implementor-Verifier) architecture divides AI workflows into three roles:

graph TD
    C["Coordinator"] -->|Task decomposition| I1["Implementor A"]
    C -->|Task decomposition| I2["Implementor B"]
    C -->|Task decomposition| I3["Implementor C"]
    I1 --> V["Verifier"]
    I2 --> V
    I3 --> V
    V -->|Pass| D["✅ Merge"]
    V -->|Fail| C

You don't need complex frameworks. In daily work, existing tools suffice:

Role	Tool	Responsibility
Coordinator	Cursor Chat / Claude Code (Plan Mode)	Analyze requirements, design solutions, decompose tasks
Implementor	Cursor Agent / Codex CLI / Aider	Implement tasks one by one per the plan
Verifier	You + test suites + linter	Review code, run tests, confirm spec compliance

Advanced technique — Adversarial verification:

After code is written, open a new AI session specifically to find bugs:

"Assume you are a security auditor. Review the following code for all potential issues:
 1. Security vulnerabilities
 2. Edge cases
 3. Performance bottlenecks
 4. Inconsistencies with project architecture

[paste code]"

This "AI reviewing AI's code" pattern is far more effective than "write and merge."

Method 5: Advanced Prompt Techniques

With tools and methodology established, a few key daily interaction techniques significantly boost AI output quality:

5.1 Role Definition — RTF Pattern (Role-Task-Format)

❌ Bad: "Write me an API"

✅ Good: "You are a senior Python backend engineer, expert in FastAPI and SQLAlchemy.
      Your task is to add CRUD APIs for the user preferences table.
      Follow the existing Repository Pattern (reference src/api/users.ts).
      Output format: First give a solution overview; I'll confirm before you write code."

The difference: Role constrains the knowledge domain, Task constrains scope, Format constrains output structure. All three are essential.

5.2 Task Chaining — Refuse One-Shot Completion

❌ Bad: "Build the entire authentication system"

✅ Good:
  Prompt 1: "Analyze the current project's auth dependencies and middleware structure"
  Prompt 2: "Based on the analysis, design a JWT authentication implementation plan"
  Prompt 3: "Implement the auth middleware (**write tests first**)"
  Prompt 4: "Implement the login API (**write tests first**)"

Each step can be independently verified, each step has a rollback point.

5.3 Plan-First — The Three-Question Method

Before letting AI write code, require it to answer three questions:

"Don't write code yet. Answer these three questions:
  1. Which files do you plan to modify?
  2. What specifically will you change in each file?
  3. What are the potential risks and edge cases?
  Wait for my confirmation before starting."

5.4 Reflective Correction — Never Say "Try Something Else"

When AI makes a mistake, most people say "That's wrong, try another approach." This is the least effective feedback. The correct way:

① Observe: "The unit test throws JSONDecodeError on line 42"
② Analyze: "It seems you didn't handle the case where input is a JSON string"
③ Instruct: "Please extend the parsing logic to support both dict and JSON string input formats"

The more precise your error feedback, the more accurate the fix. Vague feedback leads to vague fixes.

Method 6: Session Hygiene & Context Management

Core problem: Long sessions → stale information accumulates → AI gets confused → quality cliff-drops

LLM context windows aren't "bigger is better." Even with Gemini 3.1 Pro's 1M token context, model attention to information in the middle of long contexts remains lowest. More importantly — every turn's history messages consume your effective context space. 30 turns × 2000 tokens average per turn ≈ 60K tokens of historical noise, leaving less and less room for truly important information.

Solution: Session Segmentation

Session 1: Analyze requirements → produce SPEC.md → end
Session 2: Design solution based on SPEC.md → produce PLAN.md → end
Session 3: Implement tasks 1-3 based on PLAN.md → commit → end
Session 4: Implement tasks 4-6 based on PLAN.md → commit → end

Key principles:

Inject essential documents in each new session: Have AI read SPEC + PLAN at the start — 2000 tokens of distilled docs can restore full context
Summarize periodically: "Summarize what we've done so far and what's remaining," save as a file for the next session's input
The 30-turn rule: Quality likely starts degrading after 30 turns in a single session. Switch to a new one
Intermediate artifacts are relay batons: SPEC.md → PLAN.md → TASKS.md — each document is "state persistence" between sessions

Part 3: Five Anti-Patterns — Pitfall Guide

#	Anti-Pattern	Symptoms	Fix
1	Kitchen Sink	Dumping 20 files into context, AI gets more confused	Only provide the 2-3 files the current task needs
2	One-Shot Everything	"Build the entire feature," result is mediocre everywhere	Break into 5-10 atomic tasks, verify each
3	No Guardrails	AI introduces forbidden libraries or violates conventions	Explicitly list prohibitions and benchmark files in `AGENTS.md`
4	Context Rot	After 30+ turns, AI contradicts itself and forgets decisions	Segmented sessions + intermediate documents (SPEC/PLAN)
5	Merge Without Verifying	AI code "looks right," crashes in production on edge cases	TDD verification loop + adversarial code review

Why Anti-Pattern 1 happens: LLMs aren't databases — the more files you feed it, the less attention each file gets. 200 lines of precise context outperforms 5000 lines of "full dump."

Why Anti-Pattern 5 is the most dangerous: AI-generated code has a "fatal attraction" — it's well-formatted, thoroughly commented, and looks more professional than what you'd write yourself. This "surface polish" lowers your guard, while bugs hide in edge cases you wouldn't think to check (empty arrays, race conditions, timezone issues...).

If you remember only one rule: Never merge code you don't understand. Even if AI says "this is best practice."

Part 4: Tool Landscape Matrix

Methodology first, tools second. Tools serve methodology, not the other way around.

A key 2026 development is the widespread adoption of MCP (Model Context Protocol) — the USB-C of AI tools, enabling different AI coding tools to connect to databases, GitHub, file systems, and other external resources through a unified protocol. MCP support has become an important factor in tool selection.

AI-Native IDEs — Editors Rebuilt from the Ground Up for AI

Tool	Core Strength	MCP	Pricing
Cursor	AI-first editor, deep codebase indexing, layered rule system (`.cursor/rules/*.mdc`)	✅	$20/mo
Windsurf	Cost-effective, Cascade multi-step task chains	✅	$15/mo

IDE Extensions — Adding AI to Your Familiar Editor

Tool	Core Strength	Open Source
GitHub Copilot	Broadest ecosystem, Agent Mode + Coding Agent (cloud async) + MCP support	❌
Cline	Pioneer of human-in-the-loop autonomous Agents, rich MCP community	✅
Roo Code	Cline fork, role-based execution modes (Architect / Code / Debug)	✅
Continue	Fully open source, supports custom models	✅
Augment Code	Enterprise-grade semantic context engine, understands 100K+ file codebases	❌
Amazon Q Developer	Deep AWS ecosystem awareness (Lambda / CloudFormation / CDK)	❌
Gemini Code Assist	Google Cloud ecosystem integration	❌
Tabnine	Privacy-first, supports local models and enterprise custom security rules	❌

Terminal Agents (CLI) — The Fastest-Growing Category of 2025-2026

Tool	Core Strength	Open Source	Install
Claude Code	Frontier reasoning capabilities, the go-to for large-scale refactoring	❌	`npm i -g @anthropic-ai/claude-code`
Cline CLI	Rust-built, works with ChatGPT subscription, sandbox execution	✅	`npm install -g @cline/cli`
Gemini CLI	Generous free tier, 1M token context, ReAct loop	✅	`npm i -g @google/gemini-cli`
OpenCode	Model-agnostic (supports 75+ LLMs), TUI interface, privacy-first	✅	`curl -fsSL https://opencode.ai/install \
Aider	Terminal pair-programming pioneer, auto Git commit per edit	✅	{% raw %}`pip install aider-chat`
Goose	Built by Block, extensible plugin system, autonomous task execution	✅	—

Autonomous / Cloud Platform Agents — Issue → PR Fully Automated

Tool	Core Strength
GitHub Copilot Coding Agent	Based on GitHub Actions, auto-analyzes Issues, creates branches, writes code, opens PRs
Devin	Full-stack autonomous Agent, independently completes complex application development
Bolt / Lovable	Natural language → full-stack apps, ideal for rapid prototyping
Augment Intent	Multi-Agent orchestration, "living document" driven parallel development

Quick Selection Guide — One Line to Tell You What to Use

Fastest start     → GitHub Copilot (broadest ecosystem)
Strongest reasoning → Claude Code (go-to for complex refactoring)
Completely free   → Gemini CLI (1M context + free tier)
Vendor-agnostic   → OpenCode (supports 75+ models)
Git automation    → Aider (auto commit per edit)
VS Code autonomous → Cline / Roo Code
Enterprise monorepos → Augment Code
Async background  → GitHub Copilot Coding Agent

2026 best practice: Don't use just one tool. Professional developers typically use Cursor/Copilot for daily coding, Claude Code for complex reasoning and large-scale refactoring, Aider for Git integration tasks, and Gemini CLI for ultra-long context analysis. The key isn't which tool you pick — it's unifying them all with the same methodology (Spec → Plan → TDD → Verify).

Complete Mastery Workflow — The Golden Path

Finally, weaving all six methods into a complete workflow:

graph TD
    A["1. Prepare"] --> B["2. Plan"]
    B --> C["3. Execute"]
    C --> D["4. Verify"]
    D --> E["5. Feedback"]
    E -.->|"Encode into AGENTS.md"| A

    A -.- A1["AGENTS.md updated<br/>Codebase lints clean"]
    B -.- B1["AI proposes plan → you review<br/>Output: PLAN.md"]
    C -.- C1["Write tests → AI implements<br/>Run tests → git commit"]
    D -.- D1["Full test suite + AI review in new session<br/>Your final review → merge"]
    E -.- E1["Encode issues into rules<br/>Record good patterns as workflows"]

The essence of this loop is the final step: the feedback loop. Every pitfall strengthens rule files; every success crystallizes into standard procedure. After dozens of iterations, your AGENTS.md becomes the entire team's "AI user manual" — when a new team member joins, AI immediately knows how to work on your project.

Final Takeaways

Remember these four principles:

"Plan before execute" is a 10x lever. Having AI explain its approach before writing code increases success rates by an order of magnitude
Tests are the ultimate mastery tool. You write tests to define "what correct means," AI's job is to make them pass — this is what AI does best
Rule files are permanent memory. Every pitfall encoded into AGENTS.md makes your AI teammate understand your project better forever
Sessions are disposable. Don't fear starting new chats. After 30 turns, switch to a fresh session, inject essential docs, and restart — quality is always better than continuing a stale conversation

One final thought: The ceiling of AI coding tools isn't model intelligence — it's your methodology for wielding them. The six methods and five anti-patterns in this article are your complete roadmap from "being used by AI" to "using AI."

Frequently Asked Questions (FAQ)

Q1: How should project-level `AGENTS.md` rules coordinate with modular `Skills`?

Keep AGENTS.md lean (under 200 lines) as the global project constitution defining architectural boundaries, styling rules, and critical safety red lines across the whole repository. In contrast, delegate specialized repetitive tasks—such as database migrations, framework-specific refactoring, or CI workflow generation—to dedicated plug-and-play skills. Learn how to construct and package these workflows in our Skills Deep Dive: Give Your AI Coding Assistant a Professional Brain.

Q2: Why does my AI coding assistant struggle to interact with external databases and local services?

Ad-hoc natural language prompts often fail because language models lack standard execution interfaces to developer infrastructure. In 2026, the industry standard is to expose databases, Git repositories, and documentation as standardized servers via the Model Context Protocol. For step-by-step setup guides and tool integration patterns, see our MCP Protocol Guide: From Concept to Production.

Q3: How do we prevent AI assistants from causing accidental regressions in massive legacy codebases?

Enforce a strict two-stage protocol: mandate a read-only "Plan Mode" where the assistant first generates a PLAN.md detailing affected files and dependency risks without touching source code. Once you review and approve the plan, execute changes incrementally in isolated sessions verified by deterministic TDD test harnesses, ensuring errors are caught before merging into the main branch.

Originally published at Nobita Talks AI on blog.llmgo.top.