DEV Community: Sopaco

Architecture Docs Always Outdated? Let Code "Grow" Them Automatically

Sopaco — Sat, 05 Sep 2026 07:53:49 +0000

Terrain — prepares the ground so agents don't have to guess where to stand.

🔗 GitHub: https://github.com/sopaco/terrain

The Tech Lead's Pain: Documentation as Technical Debt

As a Tech Lead, you've definitely experienced:

Architecture docs that took two weeks to write become unrecognizable after one iteration
Every code review requires explaining "that's not how it's actually designed anymore"
When newcomers ask "how is this system organized?" you can only answer verbally because the Wiki is long outdated
When you ask a new AI assistant to explain the system architecture, it can only guess from the code

Once architecture docs drift from code, they become liabilities—worse than having no docs at all.

Terrain's solution: let documentation emerge from code automatically, instead of being manually written from memory.

Knowledge Factory: Code as the Source of Documentation

Terrain's core is a "knowledge factory"—starting from a Git repository, it auto-generates three layers of documentation:

Git Code Repository
    │
    ├── scan ──► index.md (Project Index)
    │
    ├── pack ──► repomix.md (Source Index)
    │
    ├── context (LLM) ──► agent/context.md (Agent Macro Architecture Context)
    │
    ├── docs (ACP) ──► human/ (Five C4 Architecture Docs + Mermaid Diagrams)
    │
    └── track ──► freshness.json (Freshness Score)

Five C4 architecture documents auto-generated, covering all levels from containers to code, with Mermaid diagrams.

Why Can It Stay "Always in Sync"?

Incremental Updates Instead of Full Regeneration

Traditional documentation tools regenerate everything each time. Terrain doesn't:

Tracks Git HEAD — Precisely knows which files changed since the last scan.
Only regenerates changed portions — Changes involve a module → update corresponding C4 doc; unrelated modules → reuse existing artifacts.
Baseline ledger — Records the Git HEAD for each scan; subsequent refreshes do incremental diffs only.

This means refreshing knowledge for a large project might take seconds instead of minutes.

Freshness Scoring System

Every knowledge asset carries a freshness score, based on:

Git code change volume vs. documentation's last generation time
CodeGraph symbol graph drift detection results

When the score drops below 50, Agents automatically reduce the weight of that knowledge asset—ensuring incorrect suggestions are never made based on outdated architectural information.

The project list interface clearly displays each project's freshness score. Stale assets are instantly visible.

Dual-Track Output: Human-Readable, AI-Consumable

Audience	Document Path	Format
Human Developers / Tech Leads	`.terrain/human/`	Narrative C4 docs + Mermaid diagrams
AI Coding Assistants	`.terrain/agent/context.md`	Structured architecture overview (≤ 14 KiB)
AI Source Retrieval	`.terrain/agent/repomix.md`	Repomix source packs
Business Knowledge	`.terrain/knowledge/`	Glossary and internal conventions

C4 docs cover four levels: Container → Component → Code → Dynamic, paired with Mermaid diagrams, unfolding layer by layer from macro architecture to micro implementation.

Source-First Trust Model

When documentation conflicts with code, Terrain follows clear priority:

repomix source code > CodeGraph symbol graph > context.md > human docs

This isn't empty talk—repomix packages real source code, CodeGraph indexes real symbol relationships. Architecture docs are just "visual interpretations" of source code. When interpretation conflicts with source, source code is always right.

Four-Phase SDD: Making Design Reviewable

Terrain's SDD (Specification-Driven Development) workflow divides development into four phases, each producing reviewable Markdown artifacts:

Phase	Output	Execution Engine
Requirements Analysis	`1.requirements.md`	Native LLM
Technical Design	`2.tech-design.md`	Native LLM
Code Generation	`3.implementation.md` + repo changes	ACP Agent
Code Review	`4.code-review.md`	Native LLM

SDD four-phase workflow. Each phase produces reviewable Markdown artifacts that Tech Leads can examine step by step.

This means Tech Leads can:

Review whether requirements specifications are accurate
Examine whether technical design is reasonable
Confirm whether code generation matches the design
Verify whether code review is thorough

Every step is documented and traceable. Every step is reviewable.

Quick Start

# Initialize a project (auto-generate C4 docs and Agent context)
terrain init ./my-repo

# View architecture overview
terrain project overview --project my-repo

# Run SDD requirements analysis phase
terrain sdd run --project my-repo --phase requirements

# Refresh knowledge (incremental, only update changed parts)
terrain refresh ./my-repo

Who Is This For?

Tech Leads — Architecture docs never go stale, always in sync with code.
Architects — C4 docs auto-generated from code, eliminating massive maintenance overhead.
Team Managers — New member onboarding compressed from days to minutes.
Code Reviewers — SDD workflow makes every design step traceable and reviewable.

"Great architecture isn't written—it grows from code."

Knowledge Management in CI/CD: How Terrain Automates Document Updates

Sopaco — Sat, 05 Sep 2026 07:51:13 +0000

Terrain — prepares the ground so agents don't have to guess where to stand.

🔗 GitHub: https://github.com/sopaco/terrain

The Real Problem in CI/CD Pipelines

Teams often face these issues in CI pipelines:

Regenerate documents on every merge—but only a few files actually changed
Knowledge assets drift from code unnoticed—until an AI assistant gives wrong suggestions based on stale information
Every CI environment requires manual toolchain installation—CodeGraph, RTK, Skills configured repeatedly
Pipeline output is hard to integrate into Agent workflows—information needs conversion to Agent-understandable formats

Terrain's design philosophy is built for automation: JSON output, incremental refresh, headless operation, one-click toolchain deployment.

Core Capability: CLI-First, JSON Everywhere

All Terrain commands are designed to be callable directly in scripts and pipelines:

JSON standard output — Every terrain tools command outputs JSON, no custom format parsing needed.
NDJSON event streams — terrain ask query --stream outputs line-by-line JSON events for real-time streaming consumption.
Headless operation — CLI doesn't depend on any display service, runs in pure terminal environments.

Typical CI Usage: Auto-Refresh Knowledge on Merge

# In CI script: auto-refresh knowledge assets after merge
terrain refresh .

# Output project freshness to logs
terrain project freshness-cached --project my-repo

# If freshness is below threshold, mark as warning
terrain tools freshness --project my-repo | jq '.score'

This means after every code merge, knowledge assets update automatically with no manual intervention. New team members who clone the repository see everything up-to-date.

Environment Standardization: `terrain env apply` One-Click Deploy

In CI or new environments, one command installs all Agent toolchains:

# Preview components to be installed
terrain env plan

# One-click install: Skills, CodeGraph, RTK, AGENTS.md snippets
terrain env apply

# Verify installation status
terrain env status

Component	Description
Skills	Standardized workflow instructions (knowledge queries, SDD, Ask, architecture analysis)
CodeGraph	Symbol call graph
RTK	Compresses shell output, saves tokens
AGENTS.md	Unified project convention snippets

Terrain's environment configuration interface. One click deploys standardized toolchains for all Agents.

Cross-Platform Distribution: npm + Pre-compiled Installers

Method	Use Case	Installation
npm package	CI/CD, headless servers, Agent pipelines	`npm install -g @terrain-ai/cli`
Pre-compiled installer	Local development, desktop use	Download from GitHub Releases
Node.js shim	Tool calls in npm environments	Auto-installed

macOS (Apple Silicon) and Windows x64 both have pre-compiled binaries
@terrain-ai/cli and @terrain-ai/rtk are both installable globally via npm
Desktop app is packaged via Tauri, includes CLI—no extra installation needed

Core Technology: Why Is It Pipeline-Friendly?

Native Rust Core, Runs Offline

All core computation is handled by terrain-core (pure Rust):

No runtime dependencies — Single binary, no dependency on Node.js/Python/databases.
Offline execution — scan, pack, search, freshness don't call LLMs.
Deterministic output — Same input produces same JSON output, suitable for automated assertions.

Incremental Refresh Engine

graph TD
    Git[Git Code Repository] --> Scan[ProjectScanner<br/>Collect Git Metadata]
    Scan --> Changed{Which Files Changed?}
    Changed -->|Changed Files| Repack[repomix Repack]
    Changed -->|Changed Modules| Update[Update Corresponding C4 Docs]
    Changed -->|No Changes| Skip[Skip Document Generation]
    Repack --> Context[Update context.md]
    Update --> Score[Recalculate Freshness Score]
    Score --> Output[Output JSON Result]

    style Skip fill:#d4f4e2,stroke:#2a9

Only processes what changed—this is the core difference between incremental refresh and traditional full regeneration. For a 100K-line project, if only a few files are modified, refresh might take just seconds.

Pipeline-Friendly Output Format

# JSON output can be processed directly by jq/scripts
terrain tools read-context --project my-repo | jq '.modules[].name'

# NDJSON stream can be consumed in real-time
terrain ask query "How does the system handle requests?" --project my-repo --stream | while read line; do
    echo "$line" | jq '.type'
done

# Suitable for CI logs and assertions
terrain project freshness-cached --project my-repo > freshness.json

Complete CI Example

#!/bin/bash
# .github/workflows/terrain-knowledge.yml

name: Update Knowledge Assets
on: [push, pull_request]

jobs:
  refresh-knowledge:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Terrain CLI
        run: npm install -g @terrain-ai/cli

      - name: Refresh knowledge assets
        run: |
          terrain refresh .
          terrain project freshness-cached --project my-repo

      - name: Check freshness threshold
        run: |
          SCORE=$(terrain project overview --project my-repo | jq '.freshness_score')
          if [ "$SCORE" -lt 50 ]; then
            echo "::warning::Knowledge assets are stale (score: $SCORE)"
          fi

Who Is This For?

DevOps Engineers — Integrate knowledge asset updates into CI pipelines.
Platform Teams — Standardize Agent environments across all projects.
Large-scale Teams — New repositories automatically get knowledge assets, no manual configuration.
ACP Integrators — Connect terrain tools JSON API to automated Agent loops.
Open Source Maintainers — Let contributors clone and immediately have full project knowledge.

"JSON output, incremental refresh, one-click deploy—a knowledge pipeline built for automation."

Struggling to Onboard New Projects? Terrain Gets You Up to Speed in 5 Minutes

Sopaco — Sat, 05 Sep 2026 07:49:38 +0000

Terrain — prepares the ground so agents don't have to guess where to stand.

🔗 GitHub: https://github.com/sopaco/terrain

The Pain We All Know

"Who owns this module? Where's the auth logic? How is the database schema designed?"

When inheriting a new project, the traditional answers are: check the Wiki (probably outdated), ask someone on Slack (might have left), or—grep through hundreds of source files blindly.

Statistics show that onboarding a new codebase typically takes developers days. And if you need an AI coding assistant to understand the project, it can only blindly grep the real-time repository, starting from scratch every time—inefficient and hallucination-prone.

Terrain compresses those days into minutes.

Terrain's project list interface, with each project showing a "freshness score"—instantly assess whether your knowledge is in sync with the code.

Core Scenario: Register → Initialize → Start Working

git clone project → terrain init → get full knowledge assets → start working

Just three steps:

Register the repo — Point to a local Git repository; Terrain registers it in the local management table.
Run initialization — Terrain automatically scans code, packages source indexes (repomix), generates C4 architecture docs and Agent context.
Start working immediately — Browse docs in the built-in reader, or ask questions directly via DeepWiki.

Five auto-generated C4 architecture documents, capturing system structure from multiple levels. Read by humans, consumed by Agents.

What Problems Does It Solve?

Without Terrain	With Terrain
Architecture knowledge scattered across Wiki, Slack, and senior engineers' heads	Engineering knowledge assets auto-generated from actual code
Documentation drifts from code after every refactoring	Incremental updates + freshness tracking; knowledge flows with Git branches
AI assistants blindly grep real-time repositories	Agent reads `context.md` first, then targeted source slices
Every team reinvents "how to get AI to understand our repo"	One-click install of Skills, CodeGraph, RTK, and `AGENTS.md`

Core Technology: Why "Always in Sync"?

Incremental Update Mechanism

Terrain doesn't regenerate everything each time. Instead, it tracks Git HEAD and only regenerates what changed since the last scan. This relies on:

Git metadata scanning — Precisely identifies which files changed.
Freshness scoring system — Each asset carries an independent score; below threshold, Agents automatically reduce its weight.
Baseline ledger — Records the Git HEAD for each scan; subsequent refreshes do incremental diffs only.

Three-Layer Retrieval Architecture

When DeepWiki or terrain tools receives a question, retrieval happens in three layers:

Layer	Asset	Retrieval Method
Macro	`agent/context.md`	Preloaded, ≤ 14 KiB architecture overview
Meso	`human/`, `knowledge/`	Full-text search, read on demand
Micro	`agent/repomix.md`	grep-pack → read-pack-file for precise location

When sources conflict, priority is: repomix source code > CodeGraph > context.md > human docs. This ensures Agents always base answers on the most reliable source information.

DeepWiki Q&A interface with precise citation sources. Knowledge has freshness guarantees—no more "making things up."

Native Rust Core: Works Offline

All core computation is handled by terrain-core (pure Rust):

Single binary, no runtime, no database — No dependency on Node.js, Python, or any database.
Fully offline execution — scan/pack/search/freshness don't call LLMs.
Cross-platform pre-compiled binaries — Ready-to-use installers for macOS (Apple Silicon) and Windows x64.

LLMs only intervene when generating context documents and DeepWiki Q&A, supporting OpenAI-compatible APIs and Ollama local deployment.

Quick Start

# Install CLI
npm install -g @terrain-ai/cli

# Register and initialize a project
terrain init ./my-repo

# Explore project knowledge
terrain search "authentication flow"
terrain ask query "How does the system handle requests?" --project my-repo

# View project overview and freshness
terrain project overview --project my-repo

Who Is This For?

Developers inheriting new projects — Say goodbye to blind file searching.
Team Tech Leads — Architecture docs stay close to code, never drift.
Anyone needing AI to understand a codebase — Give AI a reliable knowledge map instead of letting it grep from scratch.

"Terrain paves the way for Agents so they don't have to guess where to stand."

Your AI Coding Assistant Keeps Guessing Wrong? Terrain Puts It on Solid Ground

Sopaco — Sat, 05 Sep 2026 07:47:46 +0000

Terrain — prepares the ground so agents don't have to guess where to stand.

🔗 GitHub: https://github.com/sopaco/terrain

Sound Familiar?

You inherit a new project and open hundreds of files, blindly searching for architectural information. You ask an AI assistant to help understand the codebase, but it can only grep through the repository in real-time. After every refactoring, the Wiki documentation drifts from reality—and nobody notices.

Terrain was built to solve exactly these problems.

It's an engineering environment management platform for both human developers and AI coding assistants. Point it at a Git repository, and Terrain automatically scans the code, generates architectural documentation, packages source code indexes, and builds a knowledge system—so every user (human or AI) can understand the entire project in minutes instead of days.

Above: Terrain's desktop project list interface. Each project displays a freshness score, giving you instant visibility into whether your knowledge is current.

Three Core Pillars

Pillar	Metaphor	What You Get
Knowledge Assets	🗺️ Map	C4 architecture docs and Agent context auto-generated from code, always in sync
Agent Toolchain	🛤️ Road	CodeGraph, RTK, Terrain CLI—one command to install everything
Conventions & Workflows	📍 Signpost	Skills, `AGENTS.md`, four-phase SDD workflow—keeps AI working by the same rules

One Codebase, Two Audiences

Terrain's core philosophy: the same knowledge serves both humans and AI.

Audience	Path	Format
Human Developers	`.terrain/human/`	Narrative C4 docs with Mermaid diagrams
AI Coding Assistants	`.terrain/agent/context.md`	Structured architecture overview (≤ 14 KiB)
AI Source Retrieval	`.terrain/agent/repomix.md`	Repomix source packs—grep on demand, not preloaded
Business Knowledge	`.terrain/knowledge/`	Business glossary and internal conventions

Knowledge Factory: Starting from a Git repository, through scanning, packing, context generation, and documentation output—producing dual-track knowledge assets.

Why Terrain?

🕐 Minutes-level onboarding — Register repo → run init → get full C4 docs and Agent-ready context. Compress days of familiarization into minutes.
🔄 Incremental updates — Tracks Git HEAD, regenerates only changed portions, not full rebuilds.
📊 Freshness scoring — Every knowledge asset carries a score. Agents automatically reduce the weight of stale information.
🤝 Unified Agent contract — Claude Code, Codex, OpenCode, Cursor all access knowledge through the same terrain tools interface.
⚡ Native Rust performance — Single binary, no runtime, no database. scan/pack/search/freshness all run offline.
🧩 One-click toolchain deployment — terrain env apply installs CodeGraph, RTK, and preset Skills in dependency order.

Architecture at a Glance

Key Architecture Points:

terrain-core is the domain core—no LLM execution. It handles scanning, packing, searching, freshness scoring, and other pure computation tasks.
terrain-agent is the execution layer, orchestrating DeepWiki Q&A, knowledge generation, and SDD workflows.
Lightweight tasks go through native LLM (ADK Runner); heavy tool calls go through ACP subprocesses.
All Rust types auto-generate TypeScript frontend types via ts-rs, ensuring zero drift in IPC contracts.

Get Started

# Pre-compiled installers (recommended)
# Download from GitHub Releases for macOS / Windows

# Or install CLI via npm
npm install -g @terrain-ai/cli

# Register and initialize a project
terrain init ./my-repo

# Query the knowledge base
terrain ask query "How does the authentication flow work?" --project my-repo

Community & Ecosystem

Terrain evolved from Litho/deepwiki-rs (1.7k★), battle-tested at scale. It extends Litho's documentation capabilities into a full engineering environment platform—adding environment standardization, workflow management, and Agent bridging.

MIT licensed. Contributions welcome!

"Terrain prepares the ground so agents don't have to guess where to stand."

In the Age of AI Coding, What We Lack Isn't a Stronger Model — It's Ground for Agents to Stand On

Sopaco — Sat, 15 Aug 2026 09:18:40 +0000

⭐ Terrain open source: https://github.com/sopaco/terrain (MIT License) — a high-performance open-source engineering environment that lays out a "map + roads + trail markers" for AI agents. Star / Issue welcome

Over the past two years, we've watched AI coding evolve from "help me write a function" to "take over an entire requirement." Models keep getting smarter and tools keep getting stronger — but one problem has never been solved:

The moment an Agent enters an unfamiliar codebase, it instantly regresses into a "rookie intern."

It doesn't know what your architecture looks like, what your business jargon means, or which conventions it must follow. So it can only blind-grep the repo, read piles of irrelevant files, burn a huge number of tokens, and finally produce a plan that "looks right but is actually a landmine."

This isn't the model's fault. It's a problem of terrain.

A real pain point: Agents running "naked"

Drop any coding agent into your project and this is what it goes through:

🕵️ Blind-man-and-the-elephant exploration — greps from scratch, piecing together your architecture by guessing
🧠 Background knowledge only if you feed it — business context, module responsibilities, and design constraints have to be written into prompts over and over
📄 Docs always one step behind — Wiki pages and comments go stale after every refactor, so the agent may be reading the "previous architecture"
🔁 Every team reinvents the wheel — there's no standard answer to "how do I onboard AI to our repo"; everyone fends for themselves

The root problem isn't that "the model isn't smart enough" — it's that "the engineering environment isn't prepared."
We ask agents to act precisely on unfamiliar ground, yet we hand them no map at all.

That's why Terrain was born. Its core idea fits in one sentence:

Terrain prepares the ground so agents don't have to guess where to stand.
— Terrain paves the way so agents don't have to move forward in the dark.

What Terrain is: an "engineering environment," not just another "AI tool"

Terrain is a standardized, AI-friendly engineering environment management platform. Register a Git repo with it, and it automatically turns that repo into a territory where agents can land and work directly.

Its positioning breaks down into three pillars, which I'll explain with three metaphors:

Pillar	Metaphor	What you get
Engineering knowledge assets	🗺️ Map	Architecture docs and agent context auto-generated from your code and kept in sync
Standardized AI environment	🛣️ Roads	A shared "knowledge contract" (Skills, `AGENTS.md`, CLI) so every agent reads your project the same way
Development workflow	🧭 Trail markers	A four-phase standard flow (SDD) from requirements to code review, every step reviewable

Together they form a complete loop: from code to knowledge, from knowledge to action.

Three key design decisions you can read off this diagram:

Knowledge lives in the repo, not in a cloud database — .terrain/ travels with Git branches; every branch carries its own docs; knowledge follows the code;
Humans and agents consume the same knowledge contract — the same asset serves both developers and agents, so you never maintain two sources of truth;
Heavy lifting goes to external agents — Terrain doesn't do the heavy work; it delegates deep tool calls (code generation, etc.) to external coding agents over the ACP protocol.

Dual-track knowledge: one factory, two languages

The most counterintuitive — and most valuable — thing about Terrain is that it treats "docs for humans" and "context for agents" as two outputs of the same pipeline:

Audience	Path	Format
Humans	`.terrain/human/`	Narrative C4 architecture docs with Mermaid diagrams
AI agents	`.terrain/agent/context.md`	Highly compressed, structured architecture context (≤ 14 KiB)
Source index	`.terrain/agent/repomix.md`	grep-friendly source pack, read on demand, never preloaded
Domain terms	`.terrain/knowledge/`	Business glossary and team conventions

For developers, this is an architecture doc that never drifts from the code; for agents, it's a "map" to read the moment they enter the repo. One action, two audiences.

A little background: not from scratch, but proven practice

Terrain's knowledge engine is a direct descendant of Litho (open-sourced as deepwiki-rs, 1.7k+ stars on GitHub). Litho proved a thesis: generate architecture docs from code, keep them in sync, and make them agent-ready — and it holds up at scale.

Walking that proven path, Terrain upgrades the idea from "a doc generator" into "a platform":

Incremental updates — instead of regenerating everything from scratch, only the changed parts are updated;
Broad language support — works out of the box with mainstream languages like Rust, TypeScript, Python, Go, Java, and C#;
Agent-native access — via the ACP protocol, Claude Code, Codex, OpenCode, Cursor — and even the red-hot DeepSeek Harness (DSH) — can all read the same knowledge directly;
Environment standardization — one command deploys Skills, the CLI toolchain, and AGENTS.md; no per-repo manual setup.

If you liked Litho's documentation capabilities, Terrain is Litho's knowledge core plus the environment, workflow, and agent bridge built around it.

Who is Terrain for?

🧑💻 Developers — want to understand an unfamiliar codebase in seconds, or give their own project a "living" architecture doc
🧑🔧 Tech leads — want an architecture doc that tracks code evolution instead of a stale draft rotting in a Wiki
👥 Teams adopting AI coding — need a shared knowledge contract so every agent understands the project consistently
🔁 CI/CD teams — regenerate knowledge assets on every merge, so the docs never fall behind

How easy is it to get started?

Use the GUI (for everyone — one-click configuration, fully integrated features)

Use the CLI (for professional developers and CI/CD scenarios)

# Register your repo
terrain init

# Generate knowledge assets + deploy the agent toolchain in one go
terrain assets
terrain env apply

Or just launch the desktop app and do scanning, reading, Q&A, and environment setup from the GUI. From registration to a full knowledge base: minutes, not days.

Closing thoughts

The next bottleneck of AI coding is probably not "can the model write code" — it's "can the agent understand your project." Once the engineering environment is prepared — map, roads, and trail markers all in place — agents can evolve from "able to write code" to "able to write good code."

Terrain wants to be exactly that ground where agents can stand.

🚀 Open source: github.com/sopaco/terrain (MIT License)
⭐ If you believe "paving the ground for agents" is worth it, give us a Star.

If you're interested in topics like "how to keep knowledge assets fresh" or "how to let multiple agents share one understanding," there's more in this series.

Hermes Agent Codebase Packing Tool Usage Guide (repomix-rs High-Performance Edition)

Sopaco — Mon, 22 Jun 2026 01:59:35 +0000

This is the official recommended usage guide for repomix-rs — repomix-rs is a high-performance Rust rewriting of the original Repomix (TypeScript), fully compatible with the original usage pattern, and Faster, Safer, and better suited for AI Agent scenarios.

What is repomix-rs?

repomix-rs is a tool that packages an entire codebase into a single, AI-friendly file. It works seamlessly with all major LLM application scenarios including Hermes Agent, Claude, ChatGPT, and Gemini. Through repomix-rs, your codebase is presented to AI in a structured, token-controllable format, enabling AI to perform code reviews, documentation generation, and vulnerability audits more accurately.

Compared to the original Repomix, repomix-rs rewrites all core logic in Rust, bringing a qualitative leap in performance, security, and embedding capabilities.

Feature	repomix-rs (Rust)	Original Repomix (TypeScript)
Core Language	Rust	TypeScript
Runtime Speed	Millisecond-level (parallel file scanning)	Second-level (single-threaded Node.js)
Memory Safety	Compile-time guaranteed	Runtime checking
Built-in MCP Support	✅ Official MCP Server included	❌ Additional configuration required
Secretlint Integration	✅	✅
Tree-sitter	✅ 10 languages	✅ 10 languages
Token Counting	tiktoken-rs (`o200k_base`)	tiktoken (JS)
Remote Repository Packing	✅ git clone + cleanup	✅
Parallel Processing	rayon + tokio	No parallelism

🦀 Rust: The Infrastructure Choice for the AI Era

Have you noticed a trend? More and more developer ecosystem infrastructure is moving from TypeScript/Node.js to Rust. This is no accident — Rust's language characteristics are a perfect match for the demands of the AI era:

🔍 Why are developers choosing Rust?

Compile-time performance assurance — Zero-cost abstractions + no GC pauses
Memory safety — Compile-time elimination of BufferOverflow / Use-After-Free
Cross-platform single binary — Compile once, run anywhere
The core choice for the MCP era — The AI Agent ecosystem is forming a new technical standard

🌟 Rust Replacement Cases: From Bun to Vite

Bun Case: The Rust Replacement for Node.js

Rust-based high-performance runtime Bun uses Rust to rewrite the JavaScript engine, delivering:

Startup speed: Node.js ~2s → Bun ~0.1s
Execution speed: JavaScript ~2x → Bun ~5-10x
Memory footprint: Node.js ~600MB → Bun ~200-500MB

Bun's success proves the feasibility of Rust as a JS engine底层, and repomix-rs leverages Rust's high-performance characteristics to achieve a repomix rewrite.

Vite Case: Rust Rewriting of Frontend Build

How Vite ditched Webpack and used Rust to refactor frontend builds Webpack was the ruler of frontend builds, but it suffered from:

Slow startup: 5-30s
Memory usage: 500-2000MB
Node.js single-thread bottleneck

The Vue.js team decided to ditch Webpack and rewrite Vite in Rust:

Startup speed: 5-30s → ~0.5s
Memory usage: 500-2000MB → ~200MB
Concurrent processing: Event loop → Multi-threaded

Vite's success proves that Rust is the core choice for the next generation of frontend build engines.

Quick Start: No Installation Required, Just Run

Open your terminal, navigate to the project root directory, and execute any of the following:

# Method 1: Run with npx (no global installation needed)
npx repomix-rs .

# Method 2: Install globally then run directly
npm install -g repomix-rs
repomix .

# After the terminal outputs a result, send it to Hermes with: "Please read this first"

After execution, an output file (default repomix-output.xml) will be generated in the current directory. Drag that file into the Hermes Agent chat window and send the message: "Please read this project structure file first".

💡 Why repomix-rs?

In AI Agent scenarios like Hermes Agent, we recommend the Rust version repomix-rs over the TypeScript original:

Smaller image size: Rust binary ~15MB vs Node.js runtime 50+MB

Faster startup: Rust ~0.05s vs Node.js ~200ms

Smaller memory footprint: Rust ~50MB vs Node.js ~150MB

Better native MCP support: repomix-rs includes a built-in MCP Server

This is a technology choice problem — The success of Bun and Vite has already proven:
In the AI era, Rust is the better infrastructure choice.

Remote Repository Direct Packing

No cloning required — one command to pack a remote GitHub repository:

npx repomix-rs --remote https://github.com/username/project-name

Specify a branch (safer):

npx repomix-rs --remote https://github.com/username/project-name --branch main

repomix-rs's remote packing is based on the system git command. The first run pulls the full repository snapshot.
If git is unavailable, this step is skipped with a warning and the main process continues uninterrupted.

Fine-Grained Control: Which Files to Include, Which to Exclude

Create a .repomixrc configuration file:

{
  "include": ["src/**/*", "tests/**/*", "pyproject.toml", "README.md"],
  "exclude": ["**/*.log", "**/dist/**", "**/.git/**", "node_modules/**"]
}

Enable compression (extracts function signatures, compression ratio up to 50%-90%):

npx repomix-rs --compress --remove-comments --remove-empty-lines .

Include only specific language files and ignore test directories:

npx repomix-rs --include "*.rs,*.toml,Cargo.*" --ignore "target/**,tests/**" .

Output Format Selection

repomix-rs supports four output formats, switchable via the --style parameter:

npx repomix-rs --style markdown --output output.md .
npx repomix-rs --style json --output output.json .
npx repomix-rs --style plain --output output.txt .

How to Connect to Hermes Agent (Critical Steps)

Hermes Agent does not automatically scan attachment content — it must be triggered manually. The correct process is:

Run npx repomix-rs . to generate the packed file
Drag repomix-output.xml (or .md / .txt) into the Hermes Agent chat window
Send the prompt: "Please read this project structure file first"
Wait for Hermes to reply "Context loaded" before asking specific questions

Note: Hermes only supports .md / .xml / .txt plain text files.
If you accidentally send a compressed archive or binary file, the AI side will not be able to parse it.

Running as an MCP Server (Recommended for Advanced Users)

repomix-rs includes a built-in MCP Server that can be directly embedded into any AI Agent supporting the Model Context Protocol (including Hermes Agent, Cursor, and Claude Desktop):

repomix --mcp

After startup, the following MCP tools are exposed:

Tool Name	Purpose
`pack_codebase`	Pack a local codebase directory
`pack_remote_repository`	Fetch and pack a remote Git repository
`read_repomix_output`	Read a previously generated repomix output file
`grep_repomix_output`	Search content within the output file

Cursor / Claude Desktop Configuration

Claude Desktop (macOS)

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "repomix": {
      "command": "repomix",
      "args": ["--mcp"]
    }
  }
}

Cursor

Go to Settings → MCP → Add new global MCP server:

Command: repomix
Args: --mcp

Performance Comparison: Original vs repomix-rs

Scenario	Original Repomix (Node.js)	repomix-rs (Rust)	Speedup
Small-to-medium projects (< 500 files)	~3-8 seconds	~0.3-0.8 seconds	5-10×
Medium projects (500-5,000 files)	~30-120 seconds	~2-8 seconds	15-40×
Large projects (5,000+ files)	Risk of OOM	Stable completion	Unbounded
Remote repository packing	Slow (Node.js clone)	Extremely fast (git + rayon)	10-20×

Why repomix-rs Instead of the Original Repomix?

Unmatched speed — Rust zero-cost abstractions + rayon parallelism + tokio async I/O; packing the same repository takes only 1/10th the time of the original, or even less.
Native MCP Support — One command repomix --mcp directly integrates with Hermes, Claude, Cursor, and more with no additional wrapper layer needed.
Safer dependency tree — Rust binaries require no Node.js runtime; simple deployment, low CVE risk.
Fully compatible with original CLI — Parameter names, config file formats, and output formats are nearly identical; zero learning curve when switching.
More accurate token counting — tiktoken-rs uses OpenAI's official o200k_base encoding, consistent with GPT-4o, with far less counting bias than the JS version.

Frequently Asked Questions

Q: What's the difference between npx repomix-rs and npx repomix?

A: npx repomix-rs invokes repomix-rs (Rust implementation), which is faster and more stable; npx repomix invokes the original TypeScript implementation. Their command-line parameters are largely compatible.

Q: Can repomix-rs handle my Node.js project?

A: Absolutely — it is language-agnostic. repomix-rs identifies file types through file extensions and glob rules.

Q: How do I verify that exclude rules are working?

A: After generating the output, run a grep check: grep -i "secrets\|password\|API_KEY" repomix-output.xml. If sensitive words appear, check whether your glob rules are correct (e.g., **/.env, not .env).

Q: Does repomix-rs support Windows?

A: Yes, Windows x64 precompiled binaries have been released. The npm package also covers Linux/macOS/Windows.

📢 Quick Start with repomix-rs

👉 https://github.com/sopaco/repomix-rs

This guide is adapted from the original Hermes Agent Repomix usage guide, with all content migrated to repomix-rs.

repomix-rs project: https://github.com/sopaco/repomix-rs
npm package name: repomix-rs, CLI command: repomix

repomix-rs: A Deep Dive into AI Code Context Infrastructure Built with Rus

Sopaco — Mon, 22 Jun 2026 01:26:35 +0000

Architecture Perspective — Examining repomix-rs's Design Philosophy, Crate Architecture, Data Lifecycle, and Relationship with the AI Agent Ecosystem from an Engineering Height

This document is aimed at senior engineers, architects, and technical decision-makers.

Why Do We Need Code Context Infrastructure?
repomix-rs Architecture Overview
Crate Architecture Deep Dive
Data Lifecycle: From Disk to AI Context
Core Design Decisions and Trade-offs
The Philosophy of Configuration: Layered Overrides and the Principle of Least Surprise
MCP: Turning Tools into AI-Native Capabilities
The Source of Performance: A Rust Architect's Perspective
Security Architecture: A Multi-Layer Defense System
Relationship with Mainstream AI Toolchains
Project Roadmap and Ecosystem Position

Open source, feel free to give a star 💎 GitHub 🫱: https://github.com/sopaco/repomix-rs

1. Why Do We Need Code Context Infrastructure?

The LLM "Context Window Anxiety"

Although current mainstream LLMs (Deepseek, GLM) have expanded their context windows, token costs grow linearly. A medium-sized project's complete source code often exceeds 100K tokens, surpassing the comfortable processing range of most models. Traditional solutions have structural flaws:

Solution	Problem
Manual splitting + prompt engineering	High human cost, not scalable
RAG (vector retrieval)	Loses global structure; depends on embedding quality
Copy-paste into chat	Error-prone; cannot be automated
git archive + compression	AI cannot directly consume it

repomix solves a more fundamental problem: how to transmit a codebase's structure and content in an AI-readable format, precisely, completely, and reproducibly.

Token Economy

The core constraint of AI engineering is the token budget. repomix-rs addresses three problems in a targeted way:

Precise billing — Uses tiktoken-rs (OpenAI o200k_base), fully aligned with GPT-4o billing.
Budget control — --split-output allows splitting by tokens, ensuring the context window is never exceeded.
Budget optimization — --compress (Tree-sitter) saves an average of 70% tokens without losing structural information.

2. repomix-rs Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                         AI Consumer Layer                                   │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────────────────────────┐ │
│  │  Claude      │   │  Cursor      │   │  Hermes Agent                    │ │
│  │  Desktop     │   │  IDE         │   │  Custom Agents                   │ │
│  └──────┬───────┘   └──────┬───────┘   └──────────────┬───────────────────┘ │
│         └──────────┼──────────┼──────────────────────┼────────────────────┘ │
│              MCP Protocol (JSON-RPC over stdio)                             │
│                              ▼                                             │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  repomix-mcp (MCP Server)                                              │ │
│  │  Tools: pack_codebase | pack_remote_repository                         │ │
│  │         read_repomix_output | grep_repomix_output                      │ │
│  └────────────────────────────────┬───────────────────────────────────────┘ │
│                                     │                                       │
│       ┌─────────────┐ ┌────────────┴────────────────────┐                  │
│       │repomix-cli │ │          repomix-core           │                  │
│       │(clap CLI)  │ │         (Library)               │                  │
│       └──────┬──────┘ └──────────┬──────────────────────┘                  │
│              │                   │  repomix-config                          │
│              └───────────────────┤ (Config Schema)                          │
│                                  │                                          │
│  ┌─────────────┐ ┌────────────┐ ┌─────────────┐                             │
│  │File Collector│ │ Processor  │ │ Git Intg.   │                             │
│  │(rayon par.) │ │(tree-sitter)│ │ (git CLI)   │                             │
│  └──────┬────────┘ └─────┬──────┘ └──────┬──────┘                             │
│  ┌────────┴────────────────┼──────────────┼────────────────┐                │
│  │                         ▼              ▼                ▼                │
│  │  ┌──────────┐  ┌────────────────┐  ┌──────────────┐                     │
│  │  │File System│ │ Secretlint     │  │ tiktoken-rs  │                     │
│  │  │(tokio fs) │ │ (Security)     │  │ (Tokenize)   │                     │
│  │  └──────────┘  └────────────────┘  └──────────────┘                     │
└─────────────────────────────────────────────────────────────────────────────┘

3. Crate Architecture Deep Dive

3.1 Cargo Workspace Design

repomix-rs adopts a 5-Crate Cargo Workspace architecture, aligned with Rust ecosystem best practices for layered design:

repomix-rs/
├── crates/
│   ├── repomix-core/    ← Core engine (public API)
│   ├── repomix-config/  ← Config types + default modes
│   ├── repomix-shared/  ← Cross-crate shared types
│   ├── repomix-cli/     ← CLI entry point (depends on core + config)
│   └── repomix-mcp/     ← MCP Server (depends on core + shared)
├── Cargo.toml            ← workspace root
└── README.md

3.2 Individual Crate Responsibilities

`repomix-core` (Core Engine)

This is the sole "business logic" crate, encompassing:

Module	Responsibility
`file_collector`	Recursive directory scanning; apply include/exclude rules
`processor`	File content processing (compression, comment removal, AST analysis)
`output`	Serialization for four formats (XML / MD / JSON / Plain)
`git`	Git-aware operations (change frequency analysis, diff, log)
`metrics`	Token counts, character statistics, Top-N leaderboard
`security`	Secretlint integration; suspicious file detection

Exposed Traits:

#[async_trait]
pub trait ProgressCallback: Send + Sync {
    fn on_progress(&self, msg: &str);
    fn on_complete(&self, msg: &str);
    fn on_error(&self, msg: &str);
}

pub trait FileProcessor: Send + Sync {
    async fn process(&self, file: &Path) -> Result<ProcessedFile>;
}

`repomix-config` (Configuration Schema)

Dedicated to type-safe configuration and default values:

RepomixConfig: Root config struct, derives Deserialize/Serialize
OutputConfig: Output format, path, compression options
Default ignore patterns: node_modules/, __pycache__/, .git/, etc.
Global config path resolution: ~/.repomix/repomix.config.json

`repomix-shared` (Cross-Crate Shared Types)

Holds type definitions shared across crates:

pub struct ProcessedFile {
    pub path: PathBuf,
    pub content: String,
    pub tokens: usize,
    pub chars: usize,
    pub is_suspicious: bool,
    pub compress_ratio: f64,
}

pub struct PackResult {
    pub total_files: usize,
    pub total_tokens: usize,
    pub total_characters: usize,
    pub top_files_by_tokens: Vec<FileTokenCount>,
    pub suspicious_files: Vec<SuspiciousFileResult>,
    pub skipped_files: Vec<SkippedFile>,
}

`repomix-cli` (CLI Layer)

Uses clap (derive mode) for argument parsing
#[tokio::main] async main
Advanced output formatting (progress bars, colors, JSON machine-readable output)
Contains no business logic

`repomix-mcp` (MCP Server Layer)

Uses rmcp crate (Rust MCP SDK)
JSON-RPC over stdio
Internal concurrency isolation via tokio::Mutex (prevents concurrent git clone conflicts)
Exposes 4 tools, each with a serde-structured parameter schema

3.3 Layered Dependency Diagram

repomix-mcp ─────────────► repomix-core
     ▲                        │
     │                        │
repomix-cli ────────────────┤
                             │
                      repomix-config
                             ▲
                             │
                      repomix-shared

No circular dependencies; each crate is a independently testable unit.

4. Data Lifecycle: From Disk to AI Context

File System (on disk)
        │
        │ [1] Async scan (tokio async fs + rayon par_iter)
        ▼
FileEntry { path, size, mtime }
        │
        │ [2] Include/Exclude filtering
        ▼
FilteredFileEntry
        │
        │ [3] Git info enrichment (optional, git CLI)
        ▼
GitEnrichedFile { change_count, last_commit }
        │
        │ [4] Content read
        ▼
RawFileContent
        │
        │ [5] Processing pipeline (optional)
        │     ├── tree-sitter compression
        │     ├── Comment removal
        │     └── Empty-line removal
        ▼
ProcessedFile { content, tokens, chars }
        │
        │ [6] Secretlint scan (optional)
        ▼
SecureProcessedFile { is_suspicious, suspicious_patterns? }
        │
        │ [7] Format serialization
        ▼
PackOutput { xml | markdown | json | plain }
        │
        │ [8] Written to disk
        ▼
repomix-output.{xml|md|json|txt}
        │
        │ [9] Consumed by AI Consumer
        ▼
LLM Context Window

Key Design Highlights

[2] → [3] Ordered Dependency: Filter by include/exclude rules first, then enrich with Git info. Git operations are heavy (spawns subprocesses), so executing them only on the known file set is more efficient.

[5] tree-sitter pipeline: Tree-sitter provides incremental parsing. For large files, only the changed parts are re-parsed, not the full file — a detail of performance optimization.

[7] Lazy format binding: The choice of output format is deferred to the last stage of the processing pipeline. This means all formats share the same intermediate representation ProcessedFile, making it easy to extend with new formats.

5. Core Design Decisions and Trade-offs

Decision 1: Rust

Benefit	Cost
Speed: 10–20×	Steep learning curve
Memory safety	Longer compile times
Single binary deployment	Debug complexity
MCP ecosystem alignment	Ecosystem younger than JS's

Why Rust instead of Go?

Stronger type system (Traits + generics), richer abstraction capabilities
Better WASM support (potential for future browser-side execution)
Async Rust (tokio) approaches Go's performance in I/O-bound scenarios
Natural affinity with the AI/ML ecosystem (tiktoken-rs, burn, etc.)

Decision 2: Tokio over async-std

Tokio was chosen because:

The Rust community's preferred async runtime
A more mature ecosystem
tokio::Mutex is more controllable in MCP concurrency isolation scenarios

Decision 3: Plaintext JSON Over Protocol Buffers for Configuration

Uses JSON because:

Easier for humans to edit and diff
Maintains format consistency with the original Repomix's repomix.config.json
JSON Schema can be migrated to the TypeScript ecosystem (Webpack, ESLint toolchains, etc.)

Decision 4: Git CLI Subprocess over libgit2

Calls the system git command instead of using git2 (libgit2 bindings):

Git CLI has richer behavior and covers more corner cases
Avoids libgit2 version compatibility issues
In the MCP scenario, each pack operation is an independent subprocess, naturally isolated

Trade-off: Depends on git being in PATH. Without git, functionality degrades gracefully rather than failing — this is an intentional fail-soft design.

Decision 5: Four Output Formats Instead of One

Argues against a "one format serves all" approach:

LLMs have different token efficiency across formats
XML is strongly structured but verbose
Markdown is readable but has high parsing cost
Plain is the most token-efficient but lacks metadata
JSON is suitable for programmatic consumption

6. The Philosophy of Configuration: Layered Overrides and the Principle of Least Surprise

repomix-rs's configuration system follows the Layer Cake Pattern:

┌─────────────────────────────────────────────────────┐
│  CLI Flags   (highest priority, appends, not replace)│
├─────────────────────────────────────────────────────┤
│  ./repomix.config.json   (project-level)             │
├─────────────────────────────────────────────────────┤
│  ~/.repomix/repomix.config.json                     │
│  (global user-level)                                │
├─────────────────────────────────────────────────────┤
│  Hardcoded Defaults   (in-code defaults)             │
└─────────────────────────────────────────────────────┘

The three layers merge using the append-override principle:

--include appends to existing rules, does not replace
--ignore appends to existing rules, does not replace
Inner configuration overrides outer fields of the same name

The rationale is "local config takes priority; global config provides the baseline", preventing global configuration from inadvertently polluting individual projects — consistent with the Unix philosophy of "explicit over implicit".

Alignment with `.gitignore` Design

.repomixignore syntax is fully aligned with .gitignore. This is not accidental:

Git-familiar developers have zero learning cost
Glob semantics are already "consensus" in millions of engineers' minds
Reusable tooling (e.g., ignore rules from gitignore.io)

7. MCP: Turning Tools into AI-Native Capabilities

What is Model Context Protocol (MCP)?

MCP is an open protocol championed by Anthropic, defining a standardized AI Agent ↔ Tool communication interface:

┌──────────────┐        stdio JSON-RPC        ┌──────────────┐
│  Client      │ ◄────────────────────────────►│  Server      │
│ (Claude,     │                              │ (repomix-mcp)│
│  Cursor)     │                              │              │
└──────────────┘                              └──────────────┘

The protocol layer has only two core primitives: tools/list and tools/call, but through these two primitives, powerful tool compositions can be built.

repomix-rs's Role in MCP

User question
    ▼
Claude Desktop (MCP Client)
    "I need to understand this project's auth module"
    ▼
tools/call(pack_codebase, {directory: ".", compress: true})
    ▼
repomix-mcp Server
pack_directory(".") ──► repomix-core
    Tree-sitter compression (retains only auth-related function signatures)
    ▼
Returns PackResult
    ▼
Claude Desktop injects result into context
    ▼
Claude understands project structure and answers the question

Why "Native" MCP Support Matters

The original Repomix has no MCP, meaning it is just a CLI tool. For an AI Agent to use it, it must:

Spawn a subprocess to call the CLI
Parse text output
Manage the token budget itself

repomix-rs's MCP Server turns the pack operation into an AI-native capability:

Agent issues a JSON-RPC call and receives a structured result
Result is directly injected into the Agent's workflow
No lifecycle management burden on the Agent

This design upgrades repomix-rs from "a tool" to "an infrastructure component".

8. The Source of Performance: A Rust Architect's Perspective

Bottleneck Breakdown

A single pack operation roughly has four stages:

Stage	Compute Characteristics	repomix-rs Implementation	Original Repomix
File discovery	I/O + lightweight matching	`rayon::par_iter`	Single-threaded `fs.scandir`
Content reading	I/O-intensive	`tokio::fs::read`	async fs (libuv single-threaded)
AST compression	CPU-intensive	`rayon` parallel tree-sitter	Single-threaded JS
Output writing	I/O-intensive	`tokio::fs::write`	`fs.write`

Why 10–20× Speedup?

Theoretical level:

repomix-rs employs a dual-engine architecture of Rayon data parallelism + Tokio async I/O — a design capability unique to Rust:

// Pseudo-code illustration
entries.par_iter().for_each(|entry| {
    let content = rt.block_on(tokio::fs::read(&entry.path));
    let compressed = tree_sitter_compress(&content);
    result_tx.send(ProcessedFile::from(entry, compressed)).unwrap();
});

The key point: par_iter() causes Rayon to automatically utilize all available cores, while tokio::fs::read releases the thread back to the thread pool while waiting for I/O. The original Node.js "concurrency" is cooperative concurrency based on the event loop, which cannot parallelize CPU-intensive tasks across cores — this is why the tree-sitter compression stage shows the largest gap (20×+).

Engineering level:

Optimization Technique	Effect
Memory-mapped I/O (mmap)	Reduces copying, especially for large files
Zero-copy string slicing	Tree-sitter output avoids memory allocation
Streaming output	No full buffering needed, T=O(1) memory
`Arc` shared config	Zero-copy read access to config in multi-threaded scenarios
Early filtering	Applies ignore rules before reading file contents

9. Security Architecture: A Multi-Layer Defense System

┌───────────────────────────────────────────────────────────────────────┐
│ Layer 1: Configuration Layer                                         │
│  • .repomixignore excludes known dangerous paths                    │
│  • Default excludes (node_modules, .git, etc.)                      │
├───────────────────────────────────────────────────────────────────────┤
│ Layer 2: Scanning Layer (Secretlint)                                 │
│  • Regex matching for API Keys, Tokens, private keys                │
│  • Scan results configurable: warn / exclude / ignore               │
├───────────────────────────────────────────────────────────────────────┤
│ Layer 3: Output Layer                                                │
│  • Suspicious files flagged, with pattern description attached       │
│  • Supports --exclude-suspicious for hard filtering                 │
├───────────────────────────────────────────────────────────────────────┤
│ Layer 4: Runtime Layer (Rust memory safety)                          │
│  • No buffer overflows / Use-After-Free                              │
│  • No memory leaks (RAII)                                            │
│  • No data races (Send + Sync trait constraints)                     │
└───────────────────────────────────────────────────────────────────────┘

Defense in Depth is the core principle of security design. repomix-rs does not rely on a single security mechanism; it provides protection at every layer. Rust's inclusion transforms Layer 4 from "as safe as possible" into "compile-time guaranteed safety". For a tool that processes user code, potentially encountering sensitive content, this is a qualitative leap.

10. Relationship with Mainstream AI Toolchains

Position in the AI Coding Toolchain

Developer workflow
  ├─ Code editing → IDE (VSCode / Cursor)
  ├─ Code review → LLM + repomix-rs output
  ├─ Code generation → Cursor / Copilot
  ├─ Code knowledge retrieval → RAG / Embedding
  └── Codebase context injection ─────────────────────────────┐
                                                                │
  AI Agent capability stack                                    │
  ├─ Tool invocation (Function Calling) ──────────────────────┤
  ├─ Context management (Context Management) ─────────────────┤
  │   └── repomix-rs provides structured code context         │
  ├─ Long-term memory (Memory / RAG)                          │
  └─ Autonomous execution (Agentic Workflow)                  │
                                                                │
  MCP Ecosystem                                                │
  ├─ MCP Servers: filesystem, sqlite, …                       │
  ├─ MCP Servers: repomix-rs (code context) ──────────────────┤
  └─ MCP Servers: your custom tools                           │

repomix-rs occupies the codebase context provider niche in the AI coding toolchain. Its irreplaceability stems from:

It can "see" the entire project structure (RAG cannot)
It understands code's token cost (manual organization cannot)
It can be natively consumed by AI (only achievable with MCP architecture)

Relationship with RAG: Complementary, Not Competitive

RAG (Retrieval-Augmented Generation) addresses the problem of "knowing where to look", while repomix-rs addresses the problem of "how to transmit completely":

Dimension	RAG	repomix-rs
Applicable scenario	Large knowledge base retrieval	Small-to-medium project full context
Accuracy	Depends on embedding quality	Precise and complete
Token cost	Charged by retrieved chunks	Controllable compression
Setup complexity	High (requires vector DB)	Low (single command)
Real-time	Requires index updates	Real-time pack

Best practice: Use RAG + repomix-rs together — RAG for large knowledge bases; repomix-rs for current project context.

11. Project Roadmap and Ecosystem Position

Current (v2.0) Capability Matrix

Dimension	Status
Core packing	✅ Production-ready
Language support	⚠️ 10 (extensible)
MCP Server	✅ Production-ready
Remote repository packing	✅ Production-ready
Secretlint integration	✅ Basic, configurable scope
Token calculation	✅ Precise
Performance	✅ 10–40× over original
Documentation	⚠️ Moderate
Community contributions	🔄 Growing

Future Directions

Near-term (v2.x):

More language support (tree-sitter language extensions)
Incremental packing (re-process only changed files based on last pack result)
Pluggable output formats (define your own markdown templates)
Richer MCP tools (diff against baseline, etc.)

Medium-term (v3.x):

repomix-lsp: Language Server Protocol integration for real-time code context maintenance in IDEs
Streaming MCP: chunked transfer for large repositories
Multi-repository aggregation: selective packing of monorepo sub-packages

Long-term:

repomix-rs becomes one of AI Agents' standard tools (equivalent to curl's position in the HTTP toolchain)
Deep IDE integration (VSCode extension, JetBrains plugin)
WASM sandbox: browser-side execution, no local installation required

Ecosystem Position: Why Rust?

Boldly choosing Rust to rewrite developer tools is itself a technical signal. Bun chose Rust; parts of Vite chose Rust (Rolldown was rewritten in Rust). repomix-rs stands within this trend, proving that tools that are performance-sensitive, security-sensitive, and tightly coupled with the AI ecosystem are entering Rust's golden age.

Architecture Summary

repomix-rs is not a simple "Rust port" of the original Repomix. It is a tool re-architected around AI code consumption scenarios:

Layered architecture: Crates are cleanly split with clear responsibilities
Data pipeline: Token counting, compression, and filtering compose a combinable pipeline
MCP-native: First-class AI Agent integration capability
Performance: Rayon + Tokio dual-engine, fully leveraging modern hardware
Security: Multi-layer defense + Rust memory safety baseline

Choosing repomix-rs means choosing architecture for the future.

Project Resources

GitHub: https://github.com/sopaco/repomix-rs
npm: npm install -g repomix-rs

MemClaw: Give OpenClaw a 'Super Brain', Token Cost Slashed by 91%

Sopaco — Mon, 06 Apr 2026 15:01:40 +0000

If you've used OpenClaw before, you know the feeling all too well: once a conversation ends, all the API keys, technical decisions, and project background from previous chats seem to be wiped away by an eraser. This isn't a bug in OpenClaw—it's a common dilemma faced by all LLM Agents: limited context windows, and complete memory loss when the session ends.

The common solution in the community is memory plugins like OpenViking, but have you ever wondered: Is there a solution that can remember more while also saving a huge amount of Token costs?

The answer is a resounding yes. MemClaw(Plugin For OpenClaw basded on the Cortex Memory) has burst onto the scene, scoring the highest at 68.42% in the official LoCoMo benchmark (surpassing OpenViking's 52.08%), while reducing Token consumption by 11 times compared to OpenClaw+LanceDB, and improving score efficiency per thousand Tokens by 18 times.

This isn't magic—it's the power of architecture. Let's take a closer look.

Why Does OpenClaw Need "External Memory"?

If you're a heavy user of OpenClaw, these scenarios must be familiar:

Scenario 1: Repeatedly asking for API keys

User: Call Alibaba Cloud OSS to upload a file
Agent: What is your AccessKey?
User: xxx
(Next day, new session)
User: Upload another file for me
Agent: What is your AccessKey?
User: (frustrated) Didn't I tell you yesterday...

Scenario 2: "Amnesia" after long conversations

User: My project goal is to build a B2B sales tool
(After 50 rounds of conversation, discussing various technical details)
User: Based on my goal mentioned earlier, help me design the core architecture
Agent: What was the goal you mentioned earlier?

Scenario 3: Repeating the same mistakes

User: Call sales-db-query skill, incorrect parameter format
Agent: (error)
User: Correct format is {...}
(New session)
User: Call this skill again
Agent: (same error again)

The root cause of these issues: OpenClaw's native memory system has "goldfish memory"—once the context window is full, earlier content gets squeezed out; when the session ends, all states reset to zero.

OpenViking's Solution vs Cortex Memory's Overwhelming Advantage

OpenViking indeed solves this problem by giving Agents long-term memory through "virtual file system + vector search". But MemClaw(Plugin For OpenClaw basded on the Cortex Memory) delivers a crushing blow on top of this:

Benchmark data speaks for itself

System	LoCoMo Benchmark Score	Avg Tokens/Question	Score per 1K Tokens
MemClaw	68.42%	~2,900	23.6
OpenViking + OpenClaw	52.08%	~2,769	18.8
OpenClaw + LanceDB	44.55%	~33,490	1.3
OpenClaw Native Memory	35.65%	~15,982	2.2

Key insight: MemClaw(Plugin For OpenClaw basded on the Cortex Memory) not only achieves the highest score but also dominates in Token efficiency—compared to OpenClaw+LanceDB, Token savings reach 91%, with efficiency improved by 18 times.

Why Can Cortex Memory Achieve This?

The secret lies in its three-layer memory architecture:

The problem with traditional approaches: either load everything (Token explosion) or only store summaries (loss of details).

Cortex Memory's solution: Progressive layered retrieval—first quickly filter with 100-Token summaries, then refine with 2,000-Token overviews, and finally load only the truly needed full content.

Result: Retrieving 100 memories, traditional approaches need to load 100 × full content; Cortex Memory only needs 100 × 100 Tokens (L0 layer) plus a small number of L1/L2 layers.

MemClaw: One-Click Upgrade for OpenClaw

Cortex Memory provides an out-of-the-box OpenClaw plugin—MemClaw.

Installation with just one command

openclaw plugins install @memclaw/memclaw

Extremely simple configuration

Add to openclaw.json:

{
  "plugins": {
    "entries": {
      "memclaw": {
        "enabled": true,
        "config": {
          "llmApiKey": "your-api-key",
          "embeddingApiKey": "your-api-key"
        }
      }
    }
  },
  "agents": {
    "defaults": {
      "memorySearch": { "enabled": false }  // Disable native memory
    }
  }
}

Core tools at a glance

Tool	Purpose
`cortex_search`	Layered semantic search, controllable return layer
`cortex_recall`	Retrieve memories with full context
`cortex_add_memory`	Store important information for future retrieval
`cortex_commit_session`	Commit session and trigger memory extraction
`cortex_migrate`	One-click migration of OpenClaw native memories
`cortex_maintenance`	Regular maintenance (cleanup, rebuild index)

Real-World Results: From "Forgetful" to "Elephant Memory"

Case 1: Skill invocation experience accumulation

Problem: Calling a certain skill always results in parameter errors; need to retry from scratch in every new session.

MemClaw solution:

User: Call sales-db-query skill, query national sales data
Agent: (call succeeds, MemClaw automatically records correct parameter format)
(Three days later, new session)
User: Query East China region data again
Agent: (MemClaw retrieves previous successful case, uses correct format directly)

Case 2: Long conversation goals don't get lost

Problem: After 50 rounds of conversation, the Agent forgets the originally set project goal.

MemClaw solution:

User: My project goal is to build a B2B sales tool, focusing on stability
(After 100 rounds of conversation)
User: Based on my earlier goal, design the core architecture
Agent: (MemClaw retrieves original goal, outputs solution that meets constraints)

Case 3: Cross-session memory reuse

Problem: Need to re-enter API keys in every new session.

MemClaw solution:

Session A: User enters OSS key, MemClaw stores to cortex://user/preferences/
Session B: User requests file upload, MemClaw automatically retrieves key, no need to repeat input

Technical Highlights: Why Choose Cortex Memory?

1. Rust Implementation, Maximum Performance

Compared to Node.js-based memory solutions, Cortex Memory is written in Rust:

Memory safety: No GC pauses, no memory leaks
Concurrent processing: Tokio async runtime, excellent performance under high concurrency
Low resource usage: 60%+ less memory under same load

2. Fully Localized, Data Privacy

All memories stored locally in cortex-data/ directory
Vector database uses local Qdrant (or remote)
Zero cloud dependencies, suitable for sensitive data scenarios

3. Multi-Tenant Isolation

Supports multiple isolated memory spaces:

cortex-data/
├── tenants/
│   ├── project-a/     # Memories for Project A
│   ├── project-b/     # Memories for Project B
│   └── personal/      # Personal memories

4. Rich Interface

CLI tool: cortex-mem command-line management
REST API: /api/v2/* endpoints
MCP Protocol: Supports Claude Desktop, Cursor, etc.
Web Dashboard: Svelte 5 visual management

Quick Start

1. Install dependencies

# Install Qdrant vector database
docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant

# Install MemClaw plugin
openclaw plugins install @memclaw/memclaw

2. Configure API keys

{
  "plugins": {
    "entries": {
      "memclaw": {
        "enabled": true,
        "config": {
          "llmApiBaseUrl": "https://api.openai.com/v1",
          "llmApiKey": "sk-xxx",
          "llmModel": "gpt-5-mini",
          "embeddingApiBaseUrl": "https://api.openai.com/v1",
          "embeddingApiKey": "sk-xxx",
          "embeddingModel": "text-embedding-3-small"
        }
      }
    }
  }
}

3. Start using

Restart OpenClaw Gateway, MemClaw will automatically start background services. Now your Agent possesses a "super brain".

Final Thoughts

Cortex Memory is not just a simple memory storage—it's AI Agent's cognitive infrastructure.

It solves the paradox of "memory precision vs Token cost" with its three-layer architecture, guarantees performance and stability with Rust implementation, and achieves seamless integration with OpenClaw through the MemClaw plugin.

If you're tired of Agents' "goldfish memory", if you're心疼 Token consumption, if you need a production-grade long-term memory solution—Cortex Memory is worth trying.

Project URL: https://github.com/sopaco/cortex-mem

MemClaw Plugin: examples/@memclaw/plugin

Full Documentation: litho.docs/zh

How Cowork & Coding Agent Only Modifies Files That Need Changing

Sopaco — Sun, 25 Jan 2026 10:14:32 +0000

Cowork Forge - An open-source AI multi-agent development platform, serving as both an embeddable AI Coding engine and a standalone production-grade development tool. GitHub: https://github.com/sopaco/cowork-forge

Introduction

Have you ever encountered this scenario:

Your project has been under development for some time, and suddenly the product manager runs over and says: "We need to add a 'tags' feature for users."

If you follow the traditional development process, you might need to: manually analyze which files need modification, modify the data model, update API interfaces, modify frontend pages, update test cases... and worry about whether you've missed any files.

If you use AI tools, many tools choose "full regeneration"—regenerating the entire project's code. But this brings new problems: your previously manually optimized code gets overwritten, your added comments and documentation are lost, unrelated files get modified, and you need to review all the code again.

This is the problem that "incremental code updates" aims to solve.

The core idea of incremental code updates is: intelligently identify the scope of requirement changes, only modify affected files, and preserve user custom code.

In this article, I'll explore Cowork Forge's incremental code update mechanism in depth, looking at how it analyzes change impact, generates precise update plans, and how to apply it in actual projects.

The Problem with "Full Regeneration"

Before discussing incremental updates, let's look at the problems brought by "full regeneration."

Typical Full Regeneration Workflow

When using some AI tools for code generation, the typical process is: requirement change → AI analyzes new requirements → regenerate all files → overwrite original files → user custom code lost → need to review all code again → manually restore custom code.

The problem with this workflow is: AI doesn't know which code was manually added by users and which was AI-generated, so it overwrites all files, including user custom code.

Problems with Full Regeneration

First, overwriting user custom code. This is the most serious problem. Suppose you added performance optimization code in a certain file—if AI fully regenerates, your optimization code gets overwritten. Your detailed field descriptions and constraint conditions added to the user data model are also lost. Your custom validation logic added to user business logic is also overwritten.

Second, losing comments and documentation. Your added detailed comments and documentation are also lost. These comments and documentation might contain important business logic explanations, design decision records, API usage examples, etc. Losing this information increases subsequent maintenance difficulty.

Third, modifying unrelated files. Full regeneration might modify some unrelated files, increasing unnecessary risk. For example, AI might modify a completely unchanged configuration file, causing configuration to be reset.

Fourth, need to review all code again. Even if only 10% of files truly need modification, you need to review 100% of the code, wasting a lot of time. Git diff shows a large number of changes, even though most changes are unnecessary.

Actual Case: Full Regeneration of a Feature Module

Suppose you have a user management module containing user data model, user API handlers, user route definitions, user business logic, and other files. Now you need to add a "user tags" feature.

If you use full regeneration, AI will regenerate all files, adding tag fields, tag-related APIs, tag routes, tag business logic. But the problem is, your previously added caching logic in user business logic gets overwritten, your added logging in user API handlers gets overwritten, your added field validation in user data model gets overwritten.

What's the consequence? You need to manually restore all custom code, need to retest all features, and might introduce new bugs.

If you use incremental updates, AI will analyze change impact, identify affected files, generate an incremental plan, only add tag fields, only add tag-related APIs, only add tag routes, only add tag business logic, preserve caching logic. What's the result? Custom code is preserved, only need to review changed parts, Git diff is clear and concise.

This case clearly demonstrates the advantage of incremental updates: it only modifies files that truly need modification, preserves user custom code, and greatly reduces review and repair workload.

Core Ideas of Change Impact Analysis

The core of incremental code updates is change impact analysis—identifying which files and code are affected by requirement changes.

Layers of Impact Analysis

Change impact analysis can be divided into several layers: requirements layer analysis (identify changed requirements), design layer analysis (identify changed components), implementation layer analysis (identify changed modules), file layer analysis (identify affected files), code layer analysis (identify affected code snippets).

The benefit of this layered analysis is: from macro to micro, gradually narrowing the impact scope, ensuring analysis accuracy.

Requirements layer analysis identifies new requirements, deleted requirements, modified requirements. For example, if the PRD adds a "user tags" feature, this is a new requirement.

Design layer analysis identifies components that need to be added, components that need to be modified. For example, the user data model needs to add a tags field, the user API handler needs to add tag-related interfaces.

Implementation layer analysis identifies modules that need to be added, modules that need to be modified. For example, need to add a tag management module, need to modify the user management module.

File layer analysis identifies files that need to be added, files that need to be modified. For example, need to add tag-related API files, need to modify user data model files.

Code layer analysis identifies code snippets that need modification. For example, need to add a tags field in the User struct, need to add tag-related processing logic in user API handlers.

File Dependency Relationship Construction and Analysis

The core of change impact analysis is constructing a file dependency relationship graph.

The dependency relationship graph contains nodes (representing a file) and edges (representing dependency relationships). Nodes contain file path, file type, exported content, imported content. Edges contain dependency source, dependency target, dependency type (direct import, type reference, function call, data flow).

The process of constructing a dependency relationship graph is: scan all source files, analyze each file, parse AST (Abstract Syntax Tree), extract imports and exports, add nodes, build dependency relationships.

Impact propagation analysis finds direct dependencies (files that depend on the current file) and indirect dependencies (files that depend on direct dependencies), uses breadth-first search to traverse the dependency relationship graph, calculates propagation depth.

The benefit of this dependency relationship analysis is: when a file is modified, it can quickly find all affected files, ensuring no files needing updates are missed.

API-Level Impact Propagation

Besides file-level dependencies, we also need to analyze API-level impact.

API-level impact analysis identifies API changes (add, delete, modify, rename), analyzes breaking changes, identifies all affected consumers. For example, if you modify an API signature, all code calling this API needs to be updated. AI will identify this affected code and include these changes in the incremental plan.

The benefit of this API-level impact analysis is: it can ensure API changes don't break existing callers, guaranteeing system stability.

Incremental Update Mechanism Details

After understanding change impact analysis, let's look at how Cowork Forge's incremental update mechanism works.

CodeUpdater's Working Principle

CodeUpdater is the core component responsible for incremental updates.

It contains a dependency relationship graph, code analyzer, and impact analyzer. When receiving design changes, it analyzes change impact, generates an update plan, and optimizes the update plan.

The dependency relationship graph is used to track dependencies between files, the code analyzer is used to analyze code structure, and the impact analyzer is used to analyze the scope of change impact.

Update Plan Generation

The process of generating an update plan is: sort files by dependency, generate update instructions for each file, add new files.

Sorting files by dependency ensures correct dependency relationships—if file A depends on file B, then file B should be modified first. This can be achieved through topological sorting.

Generating update instructions for each file analyzes file changes—if a file has changes, it's added to the update plan. Update instructions include file path, change type (add, modify, delete), change content.

Adding new files generates templates for each new file. Templates are generated according to the project's coding standards and conventions, ensuring new files are consistent with existing code style.

How to Preserve User Custom Code

Preserving user custom code is the core challenge of incremental updates. Cowork Forge uses the following strategies.

First, code region marking. AI-generated code adds markers, and user custom code also adds markers. This way, during incremental updates, AI can identify which code is AI-generated and which is user custom.

Second, code difference analysis. It analyzes differences between original code and new code, identifies user custom code, and generates differences. This can be achieved by comparing ASTs of two versions.

Third, code merge strategy. It analyzes original code, new code, and user code, identifies conflicts, resolves conflicts, and generates merged code. Merge strategies include: preserve user code, merge AI code, resolve conflicts.

The benefit of this design is: user custom code is preserved, AI's new code is merged, conflicts are intelligently resolved, greatly reducing user workload.

Complete Process Demonstration

Let's look at how incremental updates work through a complete case.

Scenario: Adding Tag Functionality to User Module

Suppose we have a user management module, and now need to add tag functionality.

The original user data model defines user ID, name, email, creation time, update time, and other fields, plus a user custom validation method that checks if user name is empty and if email contains @ symbol.

Requirement Change

The requirement has changed, and the PRD adds a new requirement: users can add tags to tasks for categorization and filtering.

Design Change

The design document is also updated, and the user table adds a tags field.

Incremental Update Process

The incremental update process is: detect PRD change → compare old and new versions → identify requirement differences → map affected files → generate incremental plan → HITL confirm change plan → user confirms → code executor implements changes → verification module executes tests → verification results → if passed update TodoList status, if failed error analyzer diagnoses → analyze failure cause → if planning error return to mapping affected files, if execution error local fix, if environment error environment fix.

graph LR
    A[Detect PRD Change] --> B[Compare Old and New Versions]
    B --> C[Identify Requirement Differences]
    C --> D[Map Affected Files]
    D --> E[Generate Incremental Plan]
    E --> F[HITL Confirm Change Plan]
    F --> G{User Confirms?}
    G -->|Yes| H[Code Executor Implements Changes]
    G -->|No| I[Plan Adjustment]
    I --> D
    H --> J[Verification Module Executes Tests]
    J --> K{Verification Results?}
    K -->|Passed| L[Update TodoList Status]
    K -->|Failed| M[Error Analyzer Diagnoses]
    M --> N[Analyze Failure Cause]
    N --> O{Error Type?}
    O -->|Planning Error| D
    O -->|Execution Error| P[Local Fix]
    P --> J
    O -->|Environment Error| Q[Environment Fix]
    Q --> J

    classDef process fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef decision fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef action fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px

    class A,B,C,D,E,F,H,J,L,M,N process
    class G,K,O decision
    class I,P,Q action

This flowchart shows the complete incremental update process. You can see this is an intelligent process with feedback loops—if problems occur, it intelligently analyzes the cause and takes appropriate measures.

Change Impact Analysis

AI will analyze design changes and identify affected files.

Affected files include: user data model (add tags field, preserve user custom validate method), user API handler (update API processing logic, add include_tags parameter), user route definition (might need to update routes), user business logic (update business logic, add add_tag method), database migration (add new migration file).

Generate Incremental Plan

AI will generate an incremental plan containing file updates and file creation.

File updates include: user data model (add tags field, preserve user custom validate method), user API handler (update get_user handler, add include_tags parameter), user business logic (add add_tag method).

File creation includes: database migration file (create new migration file).

HITL Confirm Change Plan

User will review the change plan, seeing change plan summary (3 files modified, 1 file added, expected impact scope medium) and detailed changes (user data model adds tags field and preserves user custom validate method, user API handler updates get_user handler and adds include_tags parameter, user business logic adds add_tag method, create new database migration file).

Updated Code

After update, the user data model adds a tags field (Option>), preserving the user custom validate method. Note: the user custom validate method is completely preserved!

Verification Results

After code update completes, the verification module executes tests. The check report shows build status success, test status passed, all 18 test cases passed, user code preserved, migration applied.

Technical Challenges and Solutions

Although incremental code updates are powerful, they also face some technical challenges.

Complexity of Dependency Relationship Construction

First, multi-language support. Different programming languages have different dependency relationships, dynamic language dependency relationships are difficult to analyze statically, advanced features like macros and templates increase analysis difficulty.

The solution is to support multi-language dependency analysis. Implement language analyzers for each language, use language-specific parsing tools. For dynamic languages, combine static analysis with runtime information.

Second, dynamic language support. For dynamic languages, combine static analysis with runtime information. Use static analyzers to analyze code structure, use runtime analyzers to collect runtime information, merge results from both analyses.

Boundary Case Identification and Handling

First, circular dependency detection. Detect circular dependencies, use depth-first search to traverse the dependency relationship graph, identify circular dependencies.

Second, conditional compilation handling. Handle conditional compilation, identify conditional compilation directives, evaluate conditions, analyze dependencies within conditional blocks.

Performance Optimization Strategies

First, incremental analysis. Only analyze changed parts, use caching, only analyze changed files. If a file is in cache, use cached results; otherwise, reanalyze and update cache.

Second, parallel analysis. Analyze multiple files in parallel, use async tasks to process multiple files in parallel, improving analysis speed.

Third, cache optimization. Use cache optimization performance, check cache, if cache hit, directly return cached results; otherwise, execute analysis and update cache.

Summary

Incremental code updates are one of Cowork Forge's core features. Through intelligent change impact analysis, it only modifies affected files and preserves user custom code.

Core Value of Incremental Updates

First, preserve user custom code. Won't overwrite user manually optimized code, preserve user-added comments and documentation, maintain code's personal style.

Second, improve development efficiency. Only modify necessary files, reduce code review workload, lower risk of introducing bugs.

Third, version control friendly. Git diff is clear and concise, change history is easy to track, code review is more efficient.

Fourth, support iterative development. Rapidly respond to requirement changes, flexibly adjust feature implementation, maintain code quality.

Applicable Scenarios and Limitations

Incremental updates are suitable for projects with frequent requirement changes, projects that need to preserve user custom code, incremental development of large projects, and multi-user collaboration projects.

But incremental updates also have limitations: projects with complex dependency relationships might have inaccurate analysis, dynamic language dependency analysis is more difficult, needs good code structure support, first-time use has some learning cost.

Future Improvement Directions

First, smarter dependency analysis. Support more programming languages, improve dynamic language analysis accuracy, support more complex code patterns.

Second, more precise change identification. Improve change impact identification precision, reduce false positives and false negatives, support more fine-grained changes.

Third, smarter code merging. Improve code merging accuracy, support more complex conflict resolution, provide better merge suggestions.

Fourth, better performance optimization. Further improve analysis speed, reduce memory usage, support ultra-large projects.

Recommendations for Developers

First, maintain good code structure. Clear module division, clear dependency relationships, consistent coding style.

Second, use code markers. Mark AI-generated code, mark user custom code, facilitating incremental update identification.

Third, regularly review change plans. Carefully review AI-generated change plans, confirm change rationality, promptly adjust inappropriate changes.

Fourth, fully leverage version control. Use Git to manage code changes, commit code regularly, facilitating rollback and backtracking.

Conclusion

Incremental code updates are an important feature of AI-driven software development. It solves the problems brought by full regeneration, making AI tools more practical and reliable.

Through intelligent change impact analysis, incremental updates can precisely identify files that need modification, preserve user custom code, and improve development efficiency.

As AI technology develops, incremental updates will become smarter and more precise, providing developers with better experience.

Future software development isn't AI completely replacing humans, but AI and humans collaborating deeply. Incremental updates are an important embodiment of this collaboration.

Related Reading:

HITL Human-AI Collaboration: Why AI Code Generation Still Needs Human Oversight

Sopaco — Sun, 25 Jan 2026 10:13:04 +0000

Introduction

Imagine this scenario: You tell AI "Help me write a user login feature," and within seconds, AI generates a large amount of code. Sounds efficient, right?

But here's the question: Is this code secure? Does it meet your requirements? Are there potential security vulnerabilities? Does it follow best practices?

If you fully trust AI and directly deploy this code to production, it could have serious consequences. This is why even with powerful AI, we still need human oversight.

HITL (Human-in-the-Loop, human-AI collaboration) isn't just a simple confirmation dialog—it's a thoughtful design philosophy that finds a balance between automation efficiency and output quality.

In this article, I'll explore HITL's application in Cowork Forge in depth, looking at why human verification is needed, how to design effective HITL mechanisms, and best practices in actual use.

The Credibility Issue of AI-Generated Content

Before discussing HITL, we need to understand why AI-generated content needs human verification.

AI's Limitations

Although Large Language Models (LLMs) excel at code generation, they still have some inherent limitations.

First, hallucination problems. AI may produce "hallucinations"—generating content that looks reasonable but is actually incorrect. In code generation, this might manifest as calling non-existent APIs or libraries, using incorrect syntax or semantics, or generating code that looks correct but has logic errors. For example, AI might generate let result = user.authenticate(password); but the authenticate method might not exist at all.

Second, insufficient context understanding. AI's context window is limited. It may not fully understand the entire project's context, leading to generated code that's inconsistent with existing code style, ignores project conventions and standards, or doesn't consider interactions with other modules. Imagine AI generating new code without knowing the project already has a common logging module, so it implements its own logging functionality, resulting in inconsistent code style.

Third, lack of domain knowledge. AI's training data is public code repositories, so it may lack expertise in specific domains, such as industry-specific compliance requirements, company internal coding standards, or specific business logic complexity. For example, when developing a financial trading system, AI might not understand PCI DSS compliance requirements, know company internal security standards, or have experience designing high-performance trading systems.

Fourth, inability to make complex decisions. Certain decisions require human judgment and experience, such as technology selection trade-offs, architecture design compromises, and security policy formulation. AI may not understand the complex considerations behind these decisions. For example, when selecting a database, AI might only consider performance factors while ignoring team technology stack, operational costs, data migration difficulty, and other multi-dimensional trade-offs.

Risks of Full Automation

If we fully trust AI and skip human verification, it could bring serious risks.

First, security vulnerabilities. AI-generated code may contain SQL injection, XSS attacks, permission bypass, sensitive information leakage, and other security vulnerabilities. Once exploited, the consequences could be disastrous. For example, AI-generated query code might not use parameterized queries, leading to SQL injection vulnerabilities.

Second, quality issues. Code quality might not meet standards, such as performance problems, poor maintainability, insufficient test coverage, improper error handling. These issues increase subsequent maintenance costs. For example, AI-generated code might not consider concurrent access, leading to performance bottlenecks.

Third, business logic errors. AI might misunderstand business requirements, leading to feature implementation that doesn't meet expectations, improper handling of edge cases, or incorrect understanding of business rules. This causes the product to fail to meet user needs. For example, AI might misunderstand the requirement "users can delete their own tasks" and think administrators can also delete users' tasks.

Fourth, compliance issues. It might violate industry regulations or company policies, such as insufficient data privacy protection, missing audit logs, improper access control. These issues could lead to legal risks. For example, AI-generated user registration functionality might not record when and how users consented to terms, violating privacy regulations like GDPR.

HITL Design Philosophy

The core philosophy of HITL is: introduce human verification at key decision points to ensure the quality and controllability of AI-generated content.

This doesn't mean humans need to participate in every stage, but rather verify at key, high-impact decision points. This design philosophy is based on balancing efficiency and quality—automate repetitive, standardized tasks, and verify key decisions and complex logic manually, maintaining high efficiency while ensuring quality.

At the same time, it reflects gradual trust building—start with small tasks to build trust in AI, gradually improve AI reliability through human verification, and gradually reduce human intervention based on trust.

More importantly, it retains control—humans retain control over key decisions, AI serves as an assistant rather than a decision-maker, and can intervene and adjust when needed.

Finally, it establishes a learning and feedback loop—collect feedback through human verification, improve AI behavior and output quality, and establish a continuous improvement mechanism.

Identifying Key Decision Points

So which stages need human verification? Not all stages require human intervention—we need to identify key decision points.

Characteristics of Key Decision Points

Whether a decision point needs human verification depends on several factors.

First, scope of impact. If this decision affects multiple modules or stages, it needs human verification; if the scope of impact is small, AI can be trusted. For example, modifying a PRD affects subsequent design, coding, testing, and other stages, so it needs human verification. But optimizing a function's internal implementation has a small scope of impact, so AI can be trusted.

Second, reversibility. If a decision is irreversible or difficult to reverse, it needs human verification; if it can be easily rolled back, AI can be trusted. For example, deleting a core database table is hard to reverse once executed, so it needs human verification. But adding a new function can be easily deleted, so AI can be trusted.

Third, complexity. If a decision involves complex logic or trade-offs, it needs human verification; if it's a simple, clear choice, AI can be trusted. For example, technology selection involves trade-offs across performance, cost, team skills, and other factors, so it needs human verification. But choosing which sorting algorithm to use can be automatically selected based on data characteristics, so AI can be trusted.

Fourth, risk level. If decision failure leads to serious consequences, it needs human verification; if the risk is controllable, AI can be trusted. For example, deploying to production might cause service interruption if it fails, so it needs human verification. But running unit tests only requires fixing bugs if they fail, so the risk is controllable and AI can be trusted.

Key Verification Points in Cowork Forge

Based on the above principles, Cowork Forge sets human verification at several key nodes.

Requirements Collection Confirmation is the first verification point. Requirements are the foundation of the entire development process—if misunderstood, all subsequent work will deviate. AI might misunderstand the user's true intent, inaccurate requirement scoping leads to development scope creep. So verification is needed to confirm whether AI correctly understands the core goal, whether the functional scope is accurate, whether user roles are correctly identified, and whether constraints are complete.

PRD Confirmation is the second verification point. The PRD is the blueprint for product development, directly affecting subsequent technical design and implementation. AI might miss important requirements or acceptance criteria, and requirement priorities might be unreasonable. So verification is needed to confirm whether functional requirements are complete, whether user stories are clear, whether acceptance criteria are testable, and whether non-functional requirements are thoroughly considered.

Design Confirmation is the third verification point. Technical design determines the system's architecture and implementation approach. AI might choose an inappropriate technology stack, and the architecture design might not match the project's actual situation. So verification is needed to confirm whether technology stack selection is reasonable, whether architecture design is feasible, whether component division is reasonable, and whether the data model is correct.

Code Plan Confirmation is the fourth verification point. The code plan determines which files will be modified. AI might miss files that need modification or include files that shouldn't be modified. So verification is needed to confirm the list of files to create, the list of files to modify, the change description for each file, and whether the change order is reasonable.

Trigger Conditions for Verification Points

HITL verification point triggers aren't fixed but dynamically determined based on conditions.

For example, in the PRD generation stage, if AI's confidence is below 0.8, human verification is triggered. In the technical design stage, if complexity reaches advanced level, human verification is triggered. In the coding stage, if confidence is below 0.7 or complexity reaches intermediate level, human verification is triggered. In the check stage, human verification usually isn't triggered because this is an automated verification process.

The benefit of this dynamic trigger mechanism is: for simple, clear tasks, human verification can be skipped to improve efficiency; for complex, uncertain tasks, human verification is mandatory to ensure quality.

Actual Case Analysis: Consequences of Wrong Decisions

Let's look at an actual case to see what happens if human verification is skipped.

Suppose you're developing a payment system. AI generates a PRD that only includes two features: users can pay with credit cards and support refund operations, but misses key security requirements like PCI DSS compliance, transaction logging and auditing, and risk control rules.

If you develop directly based on this PRD, the system might have serious security vulnerabilities, could lead to user financial loss, and might violate industry regulations. This is why human verification is so important at the PRD confirmation stage—humanly review the PRD, discover missing security requirements, and require AI to supplement relevant security requirements.

This case tells us that while AI is powerful, it may lack expertise in specific domains. In key domains like finance, healthcare, and security, human verification is particularly important.

HITL Mechanism Design Philosophy

HITL isn't just a simple "confirm" button—it's a complete interaction mechanism that needs careful design.

Not Just a Simple Confirmation Dialog

Many systems' HITL mechanisms are just a simple confirmation dialog: "AI has completed PRD generation. Continue? [Cancel] [Confirm]"

The limitations of this design are obvious: users don't know what content AI generated, can't modify the content, and lack contextual information.

Cowork Forge's HITL Design

Cowork Forge's HITL mechanism is more comprehensive.

First, content display. It displays AI output in a clear, readable format, provides sufficient contextual information, and supports multiple output formats (Markdown, JSON, etc.). For example, when displaying a PRD, it uses Markdown format and includes product overview, functional requirements list, user stories and acceptance criteria, and non-functional requirements, with clear headings and structure for each section.

Second, editing support. It allows users to directly edit AI output, supports external editor integration, and preserves edit history. For example, users can directly edit the PRD in the terminal or launch external editors like VS Code for editing. Edit history is preserved for easy user backtracking and comparison.

Third, regeneration. If users aren't satisfied, they can request AI to regenerate, can provide feedback to guide AI on how to improve. For example, users can say "This PRD is missing user tagging functionality, please add it," and AI will regenerate the PRD based on this feedback.

Fourth, interaction flow. It provides a complete interaction flow, allowing users to choose to confirm directly, need editing, request regeneration, or reject. Users can choose the most appropriate operation based on their judgment.

This flowchart shows the complete HITL interaction flow. Users have multiple choices, and each choice has clear processing logic. This design gives users full control, allowing them to choose the most appropriate operation based on their judgment.

Content Display and Interaction Experience Design

Content display has several principles.

First, clarity. Use clear headings and structure, highlight important information, use appropriate formatting (tables, lists, code blocks). For example, when displaying technology stack selection, it not only lists the selected technologies but also explains the selection rationale and lists alternatives.

Second, completeness. Display all relevant information, don't miss important details, provide contextual explanations. For example, when displaying code plans, it not only lists files to modify but also explains the change content and reason for each file.

Third, actionability. Clearly tell users what they can do, provide operation guidance, explain the consequences of different operations. For example, it clearly prompts users that they can choose to confirm and continue, edit content, regenerate, or view more details.

Interaction experience design also has several key points.

First, responsiveness. Quickly respond to user operations, provide real-time feedback, avoid long waits. For example, after users click the "edit" button, the editor launches immediately; after users finish editing, the system immediately reprocesses the content.

Second, undo capability. Support undo operations, preserve operation history, allow rollback to previous states. For example, if users edit the PRD but later realize the edit was wrong, they can undo the edit and return to the previous version.

Third, smart suggestions. Provide intelligent suggestions, predict user intent, reduce user operation steps. For example, if it detects users modified functional requirements in the PRD, it prompts whether technical design needs to be regenerated.

User Editing and Regeneration Support

Cowork Forge supports integrating external editors, allowing users to conveniently edit AI output.

The external editor integration workflow is: HITL controller writes content to a temporary file, launches external editor (like VS Code), user edits content in the editor, editor returns edited content, HITL controller submits modified content to agent, agent reprocesses, returns processing result, HITL controller displays result to user.

If users aren't satisfied with AI output, they can request regeneration. The regeneration workflow is: user requests regeneration, HITL controller requests user feedback, user provides feedback, HITL controller analyzes feedback, submits feedback to agent, agent adjusts prompt, calls LLM to regenerate, LLM returns new content, agent returns new content, HITL controller compares old and new output, displays differences to user.

This design gives users full control, allowing them to choose the most appropriate operation based on their judgment.

Best Practices in Actual Use

After understanding HITL's design principles, let's see how to efficiently perform human verification in actual use.

How to Efficiently Perform Human Verification

First, preparation. Before starting to use Cowork Forge, you need to prepare. Clarify requirements—before inputting requirements, first clarify your true requirements, write down key functional points and constraints, prepare relevant reference materials. Understand the project—understand the project's technology stack and architecture, understand the project's coding standards and conventions, understand the project's business logic. Configure environment—configure external editors (like VS Code), configure Git and version control, configure necessary development tools.

Second, verification strategy. Don't try to verify all content at once, do it in stages. Quick scan—first quickly browse AI output to understand the overall structure. Key check—focus on checking key parts like core features and security requirements. Detailed review—conduct detailed reviews of important parts. Cross-verify—compare AI output with original requirements.

Using a checklist is also a good approach. Prepare a checklist for each verification stage to ensure no important check items are missed.

Third, editing techniques. Use external editors—leverage editor syntax highlighting and auto-completion, use editor search and replace functionality, leverage editor version control integration. Preserve context—preserve original content when modifying for easy comparison, add comments explaining modification reasons, preserve modification history for easy backtracking. Progressive modification—first make large structural adjustments, then detail modifications, finally format adjustments.

Common Problems and Solutions

In actual use, you might encounter some common problems.

Problem 1: AI-generated content doesn't meet expectations

Possible causes include unclear requirement description, insufficient AI context information, or AI capability limitations.

Solutions include re-describing requirements (use clearer, more specific language to describe requirements), providing more context (provide more background information and reference materials), direct editing (manually modify AI output), providing feedback (tell AI what's wrong and request regeneration).

Problem 2: AI misses important requirements

Possible causes include incomplete requirement description, AI not considering all scenarios, or AI training data lacking similar requirements.

Solutions include supplementing requirements (manually add missing requirements), providing examples (provide similar feature examples), using checklists (use requirement checklists to ensure nothing is missed).

Problem 3: AI chooses inappropriate technology stack

Possible causes include AI not understanding project technology constraints, AI training data biased toward certain technology stacks, or AI not considering team technical capabilities.

Solutions include clarifying technology constraints (explicitly state technology constraints in requirements), providing technology stack suggestions (directly tell AI which technology stack to use), manual modification (manually modify technology stack selection).

Problem 4: AI output format doesn't meet requirements

Possible causes include AI not understanding project format standards, or AI training data format inconsistency.

Solutions include providing format templates (provide standard format templates), manually adjusting format (manually adjust output format), configuring format rules (specify format rules in configuration files).

HITL Application in Team Collaboration Scenarios

In team collaboration scenarios, HITL mechanisms need to support multi-user collaboration.

First, role division. Different team members are responsible for different verification stages: product manager responsible for requirements collection and PRD verification, technical lead responsible for technical design and code plan verification, development engineer responsible for code review.

Second, collaboration process. Requirements collection → product manager verification → PRD generation → product manager verification → technical design → technical lead verification → code planning → development engineer verification → code generation → code review → check verification.

Third, collaboration tools. Tools supporting team collaboration include comments and discussion (support adding comments on content), version management (support comparing different versions), permission control (different roles have different permissions), notification mechanism (notify relevant personnel for verification).

This sequence diagram shows the HITL process in team collaboration scenarios. Different roles are responsible for different verification stages, ensuring each stage has appropriate personnel for verification.

Summary

HITL (Human-in-the-Loop) is one of Cowork Forge's core design philosophies, finding a balance between automation efficiency and output quality.

HITL Value

First, balance between efficiency and quality. Automate repetitive tasks to improve efficiency, verify key decisions manually to ensure quality, maintaining high efficiency without sacrificing quality.

Second, gradual trust building. Start with small tasks to build trust in AI, gradually improve AI reliability through human verification, gradually reduce human intervention based on trust.

Third, retaining control. Humans retain control over key decisions, AI serves as an assistant rather than a decision-maker, can intervene and adjust when needed.

Fourth, learning and feedback. Collect feedback through human verification, improve AI behavior and output quality, establish a continuous improvement loop.

Future Automation Level Evolution

As AI technology develops, HITL mechanisms will also continue to evolve.

Short-term, optimize HITL interaction experience, improve agent confidence judgment capability, reduce unnecessary verification stages.

Medium-term, introduce automated testing and verification, establish agent behavior evaluation mechanisms, dynamically adjust verification strategies based on historical data.

Long-term, achieve highly reliable AI, reduce human verification, establish complete AI trust systems, achieve deep AI-human collaboration.

Recommendations for Users

First, don't skip human verification. Even if AI output looks perfect, review it carefully. Human verification is a key link in ensuring quality.

Second, provide clear feedback. If AI output doesn't meet expectations, provide clear feedback to help AI improve.

Third, establish verification habits. Establish systematic verification habits, use checklists to ensure no important check items are missed.

Fourth, continuous learning and improvement. Through using HITL, continuously learn and improve, increase verification efficiency and quality.

Conclusion

HITL isn't distrust of AI capability, but reasonable use of AI capability. AI is a powerful tool, but it still needs human guidance and supervision.

Through HITL mechanisms, we can enjoy the efficiency improvement brought by AI while ensuring output quality and controllability. This is the true meaning of human-AI collaboration.

Future software development isn't AI replacing humans, but AI and humans collaborating deeply to create better software together.

Related Reading:

From Idea to Code: How an AI Multi-Agent System Works Like a Team to Write Software

Sopaco — Sun, 25 Jan 2026 10:11:00 +0000

Cowork Forge - An open-source AI multi-agent development platform, serving as both an embeddable AI Coding engine and a standalone production-grade development tool. GitHub: https://github.com/sopaco/cowork-forge

Introduction

Have you ever wondered what it would be like to have a virtual development team that could understand your ideas and automatically complete the entire process from requirements analysis to code delivery?

This isn't science fiction—it's happening right now.

Think about our daily development work. Aren't there always some repetitive tasks that give us headaches? When a requirement change comes in, you have to update the requirements document, then update the design document, modify the code, and update test cases... And you have to constantly communicate with product managers, designers, and test engineers throughout these stages. Time gets consumed bit by bit on these mechanical tasks, leaving little time for truly valuable creative work.

Over the years, we've witnessed the rapid evolution of development tools. From IDE code completion, to intelligent coding assistants like GitHub Copilot, to today's AI Agents. But most of these tools still remain at the "assistance" level—they can help you write a few lines of code, but cannot understand the entire software development lifecycle.

Cowork Forge attempts to break this limitation. It's not a simple code completion tool, but a complete software development automation platform. By coordinating 7 specialized AI agents, it collaborates like a real development team, transforming your ideas step by step into production-ready code.

In this article, I'll take you deep into how Cowork Forge works, showing you how these AI agents divide and collaborate, and how it differs from existing AI development tools.

Traditional Development vs. AI Collaborative Development

Before diving into Cowork Forge, let's look at the differences between traditional development patterns and AI collaborative development patterns.

The Real Experience of Traditional Development Patterns

The traditional software development process is something we're all familiar with. The product manager holds a requirements meeting, organizes the requirements document, and writes the PRD. The technical lead receives the PRD and starts architectural design, writing technical documentation. Developers receive the design document and start writing code. Testers receive the code and write test cases, executing tests. Operations personnel receive the code that passed tests and deploy it to production.

This process sounds standardized, but in practice, there are many problems.

First, too much manual work. Every stage requires significant human involvement, and much of the work is repetitive. For example, formatting requirements documents, drawing design documents, writing test cases... While this work is important, it's truly time-consuming.

Second, high communication costs. Team members need to communicate frequently, and information can easily get distorted during transmission. A product manager describes a requirement, the technical lead understands it as something else, and what the developer implements might be yet another thing. This "telephone game" is all too common in the development process.

Third, knowledge silos. Requirements, design, and code are scattered across different people and documents, making it difficult to maintain consistency. When requirements change, you have to manually modify multiple stages—not only is it inefficient, but it's also easy to miss things.

Changes Brought by AI Collaborative Development

AI collaborative development patterns thoroughly transform this process through agent automation and collaboration.

In the requirements processing stage, AI can automatically structure your ideas and generate professional requirements documents. In the document generation stage, AI can automatically generate PRDs and design documents based on requirements. In the code generation stage, AI can plan and execute code changes. In the quality verification stage, AI can automate building, testing, and verification. In the change management stage, AI can intelligently identify the scope of impact and update only affected files.

What are the core advantages of this pattern?

First, end-to-end automation. The complete process from idea to delivery can be automated, greatly reducing manual intervention stages. You don't need to switch between multiple tools—all stage outputs are unified in storage and management.

Second, specialized division of labor. Each agent focuses on a specific domain, providing specialized output. For example, the PRD agent focuses on product requirements and outputs professional PRD documents; the design agent focuses on technical architecture and applies design methods like the C4 model; the coding agent focuses on code generation and supports incremental updates; the check agent focuses on quality verification and automates building and testing.

Third, context sharing. All agents share a unified context, ensuring consistency across all stages. When requirements change, AI can quickly identify the scope of impact and update relevant content.

Fourth, rapid response. Compared to traditional patterns, AI collaborative development responds much faster. When requirements change, there's no need to manually modify multiple stages—AI can automatically handle most of the work.

Role Division Among the 7 Agents

The core of Cowork Forge is 7 specialized AI agents, each responsible for different stages of software development. They collaborate like a real development team, each performing their own duties.

Overall Architecture

Cowork Forge adopts a layered architecture design, divided from top to bottom into the agent collaboration layer, orchestration layer, and infrastructure layer.

In this architecture, the orchestration layer is responsible for coordinating agent execution, while the infrastructure layer provides foundational capabilities including tool support, verification security, data management, and interaction control. The benefit of this layered design is clear responsibility separation—the orchestration layer focuses on process control, the agent layer focuses on business logic, and the infrastructure layer provides common capabilities.

Agent Collaboration Process

The 7 agents collaborate according to a predefined process, with each agent passing results to the next agent after completing its own task. This process isn't simple linear execution, but an intelligent process with feedback loops.

An important feature of this process is intelligent rollback. When the check stage discovers a problem, the feedback agent doesn't simply request re-execution. Instead, it intelligently analyzes the root cause of the problem and determines which stage to re-execute from. If the problem is a misunderstanding of requirements, it returns to the PRD stage; if it's a design problem, it returns to the design stage; if it's a code implementation problem, it returns to the coding stage. This intelligent rollback mechanism greatly improves problem-solving efficiency.

Responsibilities of Each Agent

Let's look at what each of these agents does.

Requirements Collection Agent is the first to take the stage. Its task is to transform your ideas—which might be just a sentence like "I want to build a task management app"—into a structured requirements specification. This is like an experienced product manager who can extract real requirements from your simple description. It needs to understand vague requirement descriptions, identify key requirement points, and even ask clarification questions through the HITL mechanism.

PRD Agent, after receiving the structured requirements specification, generates a complete Product Requirements Document. It needs to transform requirements into a professional PRD format, identify dependencies between requirements, and perform priority sorting. This is like a professional product manager who can write clear, complete product documents.

Design Agent generates technical architecture and design solutions based on the PRD. It needs to apply professional architecture models, make technology selection decisions, and apply appropriate architecture patterns. This is like a senior technical lead who can design reasonable technical solutions.

Coding Stage Agent is the most complex agent, responsible for planning and executing code changes and supporting incremental updates. It contains three sub-components internally: CodePlanner is responsible for generating code change plans, CodeExecutor is responsible for executing code generation, and CodeUpdater is responsible for supporting incremental code updates. It needs to analyze project structure, plan code changes, support incremental updates, and safely execute code.

Check Agent is responsible for verifying code quality and project build status. It needs to detect project types (supporting Rust, Python, JavaScript/TypeScript), automate building and testing, analyze errors, and calculate requirements coverage. It generates a check report containing build results, test results, error analysis, quality metrics, etc.

If the check discovers problems, the Feedback Agent takes the stage. It's responsible for analyzing user feedback and determining the scope of re-execution. It analyzes feedback content, identifies the scope of impact, and intelligently decides which stage to re-execute. It generates feedback artifacts containing change requirement analysis, re-execution stage decisions, and modification suggestions.

Finally, there's the Delivery Agent. It's responsible for generating a delivery report that summarizes the entire development process. It aggregates information from all stages to generate a complete delivery report containing an implementation summary, requirements coverage, quality metrics, follow-up recommendations, etc.

Why Do We Need Multiple Agents?

You might ask: Why can't one all-powerful AI complete all the work? There are several reasons.

First, specialized capabilities. Different stages of software development require different professional knowledge and skills. Requirements analysis requires product thinking and user perspective, technical design requires architecture experience and engineering practice, code generation requires programming language and framework knowledge, and test verification requires quality assurance experience. Through specialized division of labor, each agent can provide higher quality output in its domain.

Second, single responsibility. The single responsibility principle applies not only to code design but also to agent design. Each agent focuses on a clear responsibility, making agent behavior more predictable, errors easier to locate and fix, and agents easier to test and verify.

Third, scalability. Multi-agent architecture has good scalability. You can easily add new agents, such as a performance optimization agent, a security audit agent. Each agent can be independently optimized and upgraded, and agents can be dynamically loaded as plugins.

Fourth, parallel processing. The work of certain agents can be performed in parallel, improving overall efficiency. For example, document generation and code planning can be parallelized, and code generation for multiple files can be parallelized.

End-to-End Workflow Experience

Having discussed so much theory, let's look at how Cowork Forge completes the entire development process from idea to code through a real case.

A Real Scenario

Suppose you have an idea: "I want to build a task management REST API that supports creating tasks, updating task status, and querying task lists."

With just this simple sentence, Cowork Forge can start working.

Complete Workflow

This sequence diagram shows the complete end-to-end workflow. You can see that each agent has its own responsibilities, and they transfer data through artifacts. The HITL controller intervenes at key nodes, letting users confirm AI outputs to ensure quality and controllability.

What Actually Happens at Each Stage

Stage 1: Requirements Collection

You input the sentence "I want to build a task management REST API that supports creating tasks, updating task status, and querying task lists."

The Requirements Collection Agent starts working. It calls the LLM to structure your sentence into an IdeaSpec. It identifies that the core goal is "build a task management REST API," the functional scope includes "create tasks," "update task status," "query task lists," the user role is "regular user," and constraints include "use RESTful architecture," "support concurrent access," "data persistence."

Then through the HITL mechanism, it displays this structured requirements specification to you for confirmation. After you confirm, this stage is complete.

Stage 2: PRD Generation

After receiving the IdeaSpec, the PRD Agent starts generating the Product Requirements Document.

It generates a complete PRD containing product overview, functional requirements list, user stories and acceptance criteria, and non-functional requirements. For example, for the "create task" function, it writes the function description, inputs (task title required, task description optional, task status defaults to "pending"), and outputs (task ID and task information).

Similarly, it displays the PRD to you through the HITL mechanism for confirmation.

Stage 3: Technical Design

After receiving the PRD, the Design Agent starts generating technical architecture and design solutions.

It generates a design document containing system architecture design (using C4 model), technology stack selection recommendations, component and module division, data model design, and interface definitions. For example, it selects Rust as the development language, Actix-web as the web framework, SQLite as the database, and Diesel as the ORM. It designs the Task data model, defines the TaskStatus enum, and plans the API interfaces.

Similarly, it displays the design document to you through the HITL mechanism for confirmation.

Stage 4: Coding

After receiving the design document, the Coding Stage Agent starts planning and executing code changes.

It first analyzes the project structure and generates a code change plan. For example, it plans the files that need to be created: src/main.rs, src/models.rs, src/db.rs, src/handlers.rs, src/routes.rs.

Then it generates the actual code. For example, in src/models.rs, it generates the Task struct and NewTask struct, including all necessary fields and serialization markers.

Similarly, it displays the code plan to you through the HITL mechanism for confirmation.

Stage 5: Check

After receiving the code, the Check Agent starts verifying code quality and project build status.

It runs builds and tests, generating a check report. The report shows build status (success or failure), test results (pass or fail), error analysis, and quality metrics (code coverage, cyclomatic complexity, etc.).

Stage 6: Delivery

After receiving artifacts from all stages, the Delivery Agent generates a delivery report.

This report summarizes the entire development process, including implementation summary, requirements coverage, quality metrics, and follow-up recommendations. For example, it tells you that all functional requirements have been implemented, build status passed, test pass rate is 100%, code coverage is 85%, and suggests adding user authentication and authorization, implementing task tags and categorization features in the future.

Human Verification at Key Nodes

You may have noticed that there are several key nodes in the entire workflow that require your human verification.

Requirements collection confirmation, PRD confirmation, design confirmation, code plan confirmation—these nodes all require you to review AI outputs and confirm they're correct before continuing.

This HITL (Human-in-the-Loop) mechanism ensures the quality and controllability of AI-generated content, finding a balance between automation efficiency and human control. You don't completely lose control—instead, you still have a say at key decision points.

Differentiation from Competitors

There are already many AI development tools on the market. What makes Cowork Forge unique compared to them?

Core Capability Comparison

Let's look at the comparison between Cowork Forge and several mainstream tools.

Capability	Cowork Forge	GitHub Copilot	Cursor AI	Aider
End-to-End Workflow	✅ Complete (Idea→Delivery)	❌ Only code completion	❌ Focused on code editing	❌ Only code assistance
Multi-Agent Architecture	✅ 7 specialized agents	❌ Single model	❌ Single model	❌ Single model
PRD Generation	✅ Automated	❌ N/A	❌ N/A	❌ N/A
Technical Design	✅ C4 architecture docs	❌ N/A	❌ N/A	❌ N/A
Incremental Updates	✅ Intelligent incremental analysis	❌ N/A	❌ Limited	❌ Limited
Multi-Language Support	✅ Rust, Python, JS/TS	✅ Multiple languages	✅ Multiple languages	✅ Multiple languages
Human-AI Collaboration	✅ Key decision points	❌ N/A	❌ Limited	❌ Limited
Automated Verification	✅ Build/test integration	❌ N/A	❌ N/A	❌ N/A
Security Checks	✅ Multi-layer security	❌ N/A	❌ Basic	❌ Basic
Artifact Storage	✅ Versioned artifacts	❌ N/A	❌ N/A	❌ N/A
Open Source	✅ MIT License	❌ Proprietary	❌ Proprietary	✅ MIT License
Self-Hosted	✅ Local execution	❌ Cloud only	❌ Cloud only	✅ Local execution

This comparison table clearly shows Cowork Forge's differentiated advantages. Unlike code completion tools that only assist with writing single lines of code, Cowork Forge manages the entire software development process—from initial idea collection to final delivery. This comprehensive approach ensures consistency and traceability across all stages.

Core Differentiation Advantages

Complete Development Lifecycle is the first differentiation advantage of Cowork Forge. Unlike code completion tools that only assist with writing single lines of code, Cowork Forge manages the entire software development process—from initial idea collection to final delivery. This comprehensive approach ensures consistency and traceability across all stages. You don't need to switch between multiple tools—all stage outputs are unified in storage and management. When requirements change, you can trace the impact to all related stages.

Multi-Agent Collaboration is the second differentiation advantage. Cowork Forge's 7 specialized agents collaborate like a real development team, with each agent bringing professional expertise in its domain. The PRD agent focuses on product requirements and outputs professional PRD documents; the design agent focuses on technical architecture and applies design methods like the C4 model; the coding agent focuses on code generation and supports incremental updates; the check agent focuses on quality verification and automates building and testing. This specialized division of labor ensures output quality at each stage.

Human-AI Collaboration Verification is the third differentiation advantage. Key outputs require human confirmation before proceeding, ensuring accurate requirement capture, reasonable technical decisions, and safe code changes. This balance between automation and human control distinguishes Cowork Forge from fully autonomous tools. You can control AI outputs, avoid erroneous decisions, intervene and adjust at key nodes, and retain control over the development process.

Incremental Code Updates is the fourth differentiation advantage. When requirements or designs change, Cowork Forge intelligently identifies affected files and only updates necessary content—preserving your customizations and avoiding full regeneration. This means it won't overwrite code you've manually modified, changes are more precise, unnecessary modifications are reduced, and development efficiency is improved, especially in iterative development.

Built-in Security is the fifth differentiation advantage. Multi-layer security checks prevent dangerous command execution (rm -rf, sudo, etc.), unauthorized file system access, malicious code injection, and resource exhaustion. This means you can confidently let AI execute code operations without worrying about AI deleting important files or executing dangerous commands, providing enterprise-level security guarantees.

Applicable Scenario Analysis

Cowork Forge is best suited for scenarios including: rapid new project startup (from idea to prototype, quickly validating product concepts, automatically generating project foundation and core features), incremental feature development (adding new features to existing projects, rapid response to requirement changes), small to medium projects (small team size, need to improve development efficiency, moderate project complexity suitable for automation), and documentation-driven development (need complete PRDs, design documents, emphasizing consistency between requirements and design).

Less suitable scenarios include: highly customized complex systems (complex business logic requiring many human decisions, special technology stacks where AI lacks relevant knowledge), creative design (UI/UX design requiring human creativity and aesthetics, product innovation requiring human insight), and systems with extreme performance requirements (performance-critical code requiring deep optimization, fine-grained resource management).

Summary and Outlook

Cowork Forge represents a new paradigm for AI-driven software development automation. Through multi-agent collaboration, it achieves end-to-end automation from idea to code, while retaining human control over key decisions.

Core Value Review

Cowork Forge's core value lies in: end-to-end automation (complete process automation from requirements collection to code delivery), specialized division of labor (7 agents each performing their duties, providing specialized output), human-AI collaboration (human verification at key decision points ensuring quality and controllability), incremental updates (intelligent scope identification avoiding full regeneration), and security-first (multi-layer security checks ensuring safe code execution).

Future Evolution Directions

Cowork Forge continues to evolve, with future development directions including: distributed architecture (supporting multi-machine collaboration for complex projects, improving concurrent processing capabilities), agent learning capabilities (optimizing agent behavior based on historical sessions, personalized recommendations and optimization), multi-modal support (supporting more input forms like images, audio, supporting more output types like UI prototypes), team collaboration features (multiple users participating in the same project simultaneously, real-time collaboration and conflict resolution), and more agents (performance optimization agent, security audit agent, documentation generation agent).

Recommendations for Developers

If you're considering using Cowork Forge, here are some recommendations: start with small projects (try it on simple projects first, get familiar with the workflow, gradually apply to more complex projects), make full use of HITL (don't skip human verification steps, carefully review AI outputs at key nodes), understand agent capability boundaries (know what each agent can and cannot do, use the right tool for the right scenario), maintain code quality (don't fully rely on AI, still perform code reviews, maintain good coding habits and standards), and provide feedback (if AI output doesn't meet expectations, provide detailed feedback to help improve agent behavior).

Conclusion

AI is changing the way software is developed, and Cowork Forge is an important attempt in this transformation. It's not meant to replace developers, but to become a capable assistant helping us complete repetitive work more efficiently, allowing us to focus on more valuable creative work.

Just as IDEs won't replace developers, GitHub Copilot won't replace developers, AI Agents won't replace developers either. They're just tools that help us improve efficiency, enabling us to do more.

The future belongs to developers who can skillfully use AI tools. Let's embrace this new era together.

Related Reading:

Design and Implementation of AI Memory System Evaluation Framework: How to Scientifically Measure Memory Quality

Sopaco — Sun, 28 Dec 2025 11:34:15 +0000

Abstract

Building an AI Agent memory system is just the first step; scientifically evaluating its performance, accuracy, and reliability is equally important. Cortex Memory includes a complete evaluation framework supporting recall evaluation, effectiveness evaluation, performance evaluation, and other testing scenarios. This article provides an in-depth analysis of the evaluation framework's design philosophy, core implementation, and how to use it to validate and optimize memory system performance.

1. Problem Background: The Importance of Evaluation

1.1 Why an Evaluation Framework is Needed

The quality of a memory system directly impacts AI Agent performance, but how do we quantify this quality?

1.2 Core Evaluation Metrics

Metric Category	Specific Metrics	Description
Recall	Precision@K, Recall@K, MAP, NDCG	Measure retrieval accuracy
Effectiveness	Fact extraction accuracy, classification correctness, deduplication accuracy	Measure processing quality
Performance	Latency, throughput, memory usage	Measure system efficiency
Reliability	Error rate, availability, consistency	Measure stability

1.3 Evaluation Challenges

Lack of standards: No unified evaluation standards
Data dependency: Requires high-quality test datasets
Scenario diversity: Different application scenarios focus on different metrics
Continuous change: Re-evaluation needed after system optimization

2. Evaluation Framework Architecture

2.1 Overall Architecture

2.2 Core Components

2.2.1 Dataset Manager

pub struct DatasetManager {
    datasets: HashMap<String, Dataset>,
}

pub struct Dataset {
    pub id: String,
    pub name: String,
    pub description: String,
    pub test_cases: Vec<TestCase>,
    pub metadata: DatasetMetadata,
}

pub struct TestCase {
    pub id: String,
    pub query: String,
    pub expected_results: Vec<ExpectedResult>,
    pub metadata: TestCaseMetadata,
}

pub struct ExpectedResult {
    pub memory_id: String,
    pub relevance_score: f32,
    pub position: usize,
}

pub enum DatasetType {
    Recall,        // Recall testing
    Effectiveness, // Effectiveness testing
    Performance,   // Performance testing
    Mixed,         // Mixed testing
}

2.2.2 Evaluator Interface

#[async_trait]
pub trait Evaluator: Send + Sync {
    async fn evaluate(&self, dataset: &Dataset) -> Result<EvaluationResult>;

    fn name(&self) -> &str;

    fn metrics(&self) -> Vec<MetricDefinition>;
}

pub struct EvaluationResult {
    pub evaluator_name: String,
    pub dataset_id: String,
    pub metrics: HashMap<String, MetricValue>,
    pub details: EvaluationDetails,
    pub timestamp: DateTime<Utc>,
}

pub struct MetricValue {
    pub name: String,
    pub value: f64,
    pub unit: String,
    pub description: String,
}

pub struct EvaluationDetails {
    pub test_cases_passed: usize,
    pub test_cases_total: usize,
    pub errors: Vec<EvaluationError>,
    pub warnings: Vec<EvaluationWarning>,
}

3. Recall Evaluation

3.1 Evaluation Metrics

3.1.1 Precision@K

pub struct PrecisionAtK {
    pub k: usize,
}

impl PrecisionAtK {
    pub fn calculate(
        &self,
        retrieved: &[ScoredMemory],
        relevant: &HashSet<String>,
    ) -> f64 {
        if retrieved.is_empty() {
            return 0.0;
        }

        let top_k = retrieved.iter().take(self.k);
        let relevant_count = top_k
            .filter(|m| relevant.contains(&m.memory.id))
            .count();

        relevant_count as f64 / self.k as f64
    }
}

// Usage example
let precision_at_5 = PrecisionAtK { k: 5 };
let precision = precision_at_5.calculate(&results, &relevant_ids);

3.1.2 Recall@K

pub struct RecallAtK {
    pub k: usize,
}

impl RecallAtK {
    pub fn calculate(
        &self,
        retrieved: &[ScoredMemory],
        relevant: &HashSet<String>,
    ) -> f64 {
        if relevant.is_empty() {
            return 1.0;
        }

        let top_k = retrieved.iter().take(self.k);
        let retrieved_relevant = top_k
            .filter(|m| relevant.contains(&m.memory.id))
            .count();

        retrieved_relevant as f64 / relevant.len() as f64
    }
}

3.1.3 Mean Average Precision (MAP)

pub struct MeanAveragePrecision;

impl MeanAveragePrecision {
    pub fn calculate(
        &self,
        all_results: &[Vec<ScoredMemory>],
        all_relevant: &[HashSet<String>],
    ) -> f64 {
        let mut aps = Vec::new();

        for (results, relevant) in all_results.iter().zip(all_relevant.iter()) {
            if relevant.is_empty() {
                continue;
            }

            let ap = self.average_precision(results, relevant);
            aps.push(ap);
        }

        if aps.is_empty() {
            return 0.0;
        }

        aps.iter().sum::<f64>() / aps.len() as f64
    }

    fn average_precision(
        &self,
        results: &[ScoredMemory],
        relevant: &HashSet<String>,
    ) -> f64 {
        let mut precision_sum = 0.0;
        let mut relevant_count = 0;

        for (i, result) in results.iter().enumerate() {
            if relevant.contains(&result.memory.id) {
                relevant_count += 1;
                let precision = relevant_count as f64 / (i + 1) as f64;
                precision_sum += precision;
            }
        }

        if relevant_count == 0 {
            return 0.0;
        }

        precision_sum / relevant_count as f64
    }
}

3.1.4 Normalized Discounted Cumulative Gain (NDCG)

pub struct NDCG {
    pub k: usize,
}

impl NDCG {
    pub fn calculate(
        &self,
        retrieved: &[ScoredMemory],
        relevant: &HashMap<String, f64>,
    ) -> f64 {
        let dcg = self.dcg(retrieved, relevant);
        let idcg = self.idcg(relevant);

        if idcg == 0.0 {
            return 0.0;
        }

        dcg / idcg
    }

    fn dcg(
        &self,
        retrieved: &[ScoredMemory],
        relevant: &HashMap<String, f64>,
    ) -> f64 {
        let mut dcg = 0.0;

        for (i, result) in retrieved.iter().take(self.k).enumerate() {
            let relevance = relevant.get(&result.memory.id).unwrap_or(&0.0);
            dcg += relevance / (i + 1) as f64;
        }

        dcg
    }

    fn idcg(&self, relevant: &HashMap<String, f64>) -> f64 {
        let mut sorted_relevance: Vec<_> = relevant.values().cloned().collect();
        sorted_relevance.sort_by(|a, b| b.partial_cmp(a).unwrap());

        let mut idcg = 0.0;
        for (i, relevance) in sorted_relevance.iter().take(self.k).enumerate() {
            idcg += relevance / (i + 1) as f64;
        }

        idcg
    }
}

3.2 Recall Evaluator Implementation

pub struct RecallEvaluator {
    config: RecallEvaluatorConfig,
    memory_manager: Arc<MemoryManager>,
}

#[derive(Debug, Clone)]
pub struct RecallEvaluatorConfig {
    pub k_values: Vec<usize>,
    pub similarity_thresholds: Vec<f32>,
    pub max_results_per_query: usize,
}

#[async_trait]
impl Evaluator for RecallEvaluator {
    async fn evaluate(&self, dataset: &Dataset) -> Result<EvaluationResult> {
        let mut metrics = HashMap::new();
        let mut details = EvaluationDetails {
            test_cases_passed: 0,
            test_cases_total: dataset.test_cases.len(),
            errors: Vec::new(),
            warnings: Vec::new(),
        };

        // Calculate Precision and Recall for each K value
        for k in &self.config.k_values {
            let mut precision_values = Vec::new();
            let mut recall_values = Vec::new();

            for test_case in &dataset.test_cases {
                match self.evaluate_test_case(test_case, *k).await {
                    Ok((precision, recall)) => {
                        precision_values.push(precision);
                        recall_values.push(recall);
                        details.test_cases_passed += 1;
                    }
                    Err(e) => {
                        details.errors.push(EvaluationError {
                            test_case_id: test_case.id.clone(),
                            message: e.to_string(),
                        });
                    }
                }
            }

            // Calculate averages
            let avg_precision = precision_values.iter().sum::<f64>() / precision_values.len() as f64;
            let avg_recall = recall_values.iter().sum::<f64>() / recall_values.len() as f64;

            metrics.insert(
                format!("precision@{}", k),
                MetricValue {
                    name: format!("Precision@{}", k),
                    value: avg_precision,
                    unit: "score".to_string(),
                    description: format!("Average precision at K={}", k),
                },
            );

            metrics.insert(
                format!("recall@{}", k),
                MetricValue {
                    name: format!("Recall@{}", k),
                    value: avg_recall,
                    unit: "score".to_string(),
                    description: format!("Average recall at K={}", k),
                },
            );
        }

        // Calculate MAP and NDCG
        let all_results: Vec<Vec<ScoredMemory>> = dataset
            .test_cases
            .iter()
            .filter_map(|tc| self.get_retrieved_results(tc).await.ok())
            .collect();

        let all_relevant: Vec<HashSet<String>> = dataset
            .test_cases
            .iter()
            .map(|tc| {
                tc.expected_results
                    .iter()
                    .map(|er| er.memory_id.clone())
                    .collect()
            })
            .collect();

        let map = MeanAveragePrecision.calculate(&all_results, &all_relevant);
        metrics.insert(
            "map".to_string(),
            MetricValue {
                name: "Mean Average Precision".to_string(),
                value: map,
                unit: "score".to_string(),
                description: "Mean Average Precision across all queries".to_string(),
            },
        );

        Ok(EvaluationResult {
            evaluator_name: self.name().to_string(),
            dataset_id: dataset.id.clone(),
            metrics,
            details,
            timestamp: Utc::now(),
        })
    }

    fn name(&self) -> &str {
        "recall_evaluator"
    }

    fn metrics(&self) -> Vec<MetricDefinition> {
        vec![
            MetricDefinition {
                name: "precision@k".to_string(),
                description: "Precision at K".to_string(),
                unit: "score".to_string(),
            },
            MetricDefinition {
                name: "recall@k".to_string(),
                description: "Recall at K".to_string(),
                unit: "score".to_string(),
            },
            MetricDefinition {
                name: "map".to_string(),
                description: "Mean Average Precision".to_string(),
                unit: "score".to_string(),
            },
        ]
    }
}

impl RecallEvaluator {
    async fn evaluate_test_case(
        &self,
        test_case: &TestCase,
        k: usize,
    ) -> Result<(f64, f64)> {
        // Execute search
        let retrieved = self
            .memory_manager
            .search(&test_case.query, &Filters::default(), k)
            .await?;

        // Build relevant memories set
        let relevant: HashSet<String> = test_case
            .expected_results
            .iter()
            .map(|er| er.memory_id.clone())
            .collect();

        // Calculate Precision and Recall
        let precision = PrecisionAtK { k }.calculate(&retrieved, &relevant);
        let recall = RecallAtK { k }.calculate(&retrieved, &relevant);

        Ok((precision, recall))
    }

    async fn get_retrieved_results(
        &self,
        test_case: &TestCase,
    ) -> Result<Vec<ScoredMemory>> {
        self.memory_manager
            .search(&test_case.query, &Filters::default(), self.config.max_results_per_query)
            .await
    }
}

4. Effectiveness Evaluation

4.1 Evaluation Dimensions

4.1.1 Fact Extraction Accuracy

pub struct FactExtractionEvaluator {
    llm_client: Box<dyn LLMClient>,
}

impl FactExtractionEvaluator {
    pub async fn evaluate_extraction_accuracy(
        &self,
        conversation: &[Message],
        extracted_facts: &[ExtractedFact],
        ground_truth: &[ExtractedFact],
    ) -> Result<ExtractionAccuracy> {
        // Calculate precision
        let precision = self.calculate_precision(extracted_facts, ground_truth).await?;

        // Calculate recall
        let recall = self.calculate_recall(extracted_facts, ground_truth).await?;

        // Calculate F1 score
        let f1 = if precision + recall > 0.0 {
            2.0 * precision * recall / (precision + recall)
        } else {
            0.0
        };

        Ok(ExtractionAccuracy {
            precision,
            recall,
            f1,
            extracted_count: extracted_facts.len(),
            ground_truth_count: ground_truth.len(),
        })
    }

    async fn calculate_precision(
        &self,
        extracted: &[ExtractedFact],
        ground_truth: &[ExtractedFact],
    ) -> Result<f64> {
        if extracted.is_empty() {
            return Ok(0.0);
        }

        let mut correct_count = 0;

        for fact in extracted {
            let is_correct = self.is_fact_correct(fact, ground_truth).await?;
            if is_correct {
                correct_count += 1;
            }
        }

        Ok(correct_count as f64 / extracted.len() as f64)
    }

    async fn is_fact_correct(
        &self,
        fact: &ExtractedFact,
        ground_truth: &[ExtractedFact],
    ) -> Result<bool> {
        // Use LLM to judge if the fact is correct
        let prompt = format!(
            "Compare the following fact with the ground truth facts:\n\n\
             Fact to evaluate: {}\n\n\
             Ground truth facts:\n{}\n\n\
             Is the fact correct and present in the ground truth? (yes/no)",
            fact.content,
            ground_truth
                .iter()
                .map(|f| format!("- {}", f.content))
                .collect::<Vec<_>>()
                .join("\n")
        );

        let response = self.llm_client.complete(&prompt).await?;
        Ok(response.to_lowercase().contains("yes"))
    }
}

pub struct ExtractionAccuracy {
    pub precision: f64,
    pub recall: f64,
    pub f1: f64,
    pub extracted_count: usize,
    pub ground_truth_count: usize,
}

4.1.2 Classification Correctness

pub struct ClassificationEvaluator {
    llm_client: Box<dyn LLMClient>,
}

impl ClassificationEvaluator {
    pub async fn evaluate_classification_accuracy(
        &self,
        memories: &[(Memory, MemoryType)],
    ) -> Result<ClassificationAccuracy> {
        let mut correct_count = 0;
        let mut total_count = memories.len();

        for (memory, expected_type) in memories {
            let predicted_type = self.predict_type(&memory.content).await?;

            if predicted_type == *expected_type {
                correct_count += 1;
            }
        }

        let accuracy = correct_count as f64 / total_count as f64;

        // Calculate precision and recall for each category
        let per_class_metrics = self.calculate_per_class_metrics(memories).await?;

        Ok(ClassificationAccuracy {
            overall_accuracy: accuracy,
            per_class_metrics,
            total_count,
        })
    }

    async fn predict_type(&self, content: &str) -> Result<MemoryType> {
        let prompt = format!(
            "Classify the following memory content:\n\n\
             Content: {}\n\n\
             Classify as one of: Conversational, Procedural, Factual, Semantic, Episodic, Personal\n\n\
             Classification:",
            content
        );

        let response = self.llm_client.complete(&prompt).await?;
        Ok(MemoryType::parse(&response))
    }

    async fn calculate_per_class_metrics(
        &self,
        memories: &[(Memory, MemoryType)],
    ) -> Result<HashMap<MemoryType, ClassMetrics>> {
        let mut metrics: HashMap<MemoryType, ClassMetrics> = HashMap::new();

        for (memory, expected_type) in memories {
            let predicted_type = self.predict_type(&memory.content).await?;

            // Update expected type metrics
            metrics
                .entry(*expected_type)
                .or_insert_with(ClassMetrics::new)
                .total += 1;

            if predicted_type == *expected_type {
                metrics.get_mut(&predicted_type).unwrap().true_positives += 1;
            } else {
                metrics
                    .get_mut(&predicted_type)
                    .or_insert_with(ClassMetrics::new)
                    .false_positives += 1;
                metrics.get_mut(&expected_type).unwrap().false_negatives += 1;
            }
        }

        // Calculate precision and recall
        for (_, metric) in metrics.iter_mut() {
            metric.precision = if metric.true_positives + metric.false_positives > 0 {
                metric.true_positives as f64
                    / (metric.true_positives + metric.false_positives) as f64
            } else {
                0.0
            };

            metric.recall = if metric.true_positives + metric.false_negatives > 0 {
                metric.true_positives as f64
                    / (metric.true_positives + metric.false_negatives) as f64
            } else {
                0.0
            };

            metric.f1 = if metric.precision + metric.recall > 0.0 {
                2.0 * metric.precision * metric.recall / (metric.precision + metric.recall)
            } else {
                0.0
            };
        }

        Ok(metrics)
    }
}

pub struct ClassMetrics {
    pub true_positives: usize,
    pub false_positives: usize,
    pub false_negatives: usize,
    pub total: usize,
    pub precision: f64,
    pub recall: f64,
    pub f1: f64,
}

impl ClassMetrics {
    fn new() -> Self {
        Self {
            true_positives: 0,
            false_positives: 0,
            false_negatives: 0,
            total: 0,
            precision: 0.0,
            recall: 0.0,
            f1: 0.0,
        }
    }
}

5. Performance Evaluation

5.1 Benchmark Testing

use criterion::{black_box, criterion_group, criterion_main, Criterion, BenchmarkId};

pub struct PerformanceBenchmark {
    memory_manager: Arc<MemoryManager>,
}

impl PerformanceBenchmark {
    pub fn benchmark_search_latency(c: &mut Criterion) {
        let mut group = c.benchmark_group("search_latency");

        for dataset_size in [1000, 5000, 10000, 50000].iter() {
            let manager = self.create_test_manager(*dataset_size).await;

            group.bench_with_input(
                BenchmarkId::from_parameter(dataset_size),
                dataset_size,
                |b, _| {
                    b.iter(|| {
                        black_box(
                            tokio::runtime::Runtime::new()
                                .unwrap()
                                .block_on(manager.search("test query", &Filters::default(), 10))
                        )
                    });
                },
            );
        }

        group.finish();
    }

    pub fn benchmark_insert_latency(c: &mut Criterion) {
        let mut group = c.benchmark_group("insert_latency");

        for content_length in [100, 500, 1000, 5000].iter() {
            let manager = self.create_test_manager(0).await;
            let content = "A".repeat(*content_length);

            group.bench_with_input(
                BenchmarkId::from_parameter(content_length),
                content_length,
                |b, _| {
                    b.iter(|| {
                        black_box(
                            tokio::runtime::Runtime::new()
                                .unwrap()
                                .block_on(
                                    manager.create_memory(
                                        content.clone(),
                                        MemoryMetadata::default(),
                                    )
                                )
                        )
                    });
                },
            );
        }

        group.finish();
    }

    pub fn benchmark_throughput(c: &mut Criterion) {
        let manager = self.create_test_manager(10000).await;

        let mut group = c.benchmark_group("throughput");

        group.bench_function("concurrent_searches", |b| {
            b.iter(|| {
                let rt = tokio::runtime::Runtime::new().unwrap();

                rt.block_on(async {
                    let handles: Vec<_> = (0..100)
                        .map(|_| {
                            let manager = manager.clone();
                            tokio::spawn(async move {
                                manager.search("test query", &Filters::default(), 10).await
                            })
                        })
                        .collect();

                    futures::future::join_all(handles).await;
                });
            });
        });

        group.finish();
    }
}

criterion_group! {
    name = benches;
    config = Criterion::default().sample_size(100);
    targets = PerformanceBenchmark::benchmark_search_latency,
              PerformanceBenchmark::benchmark_insert_latency,
              PerformanceBenchmark::benchmark_throughput
}

criterion_main!(benches);

5.2 Load Testing

use load_testing::LoadTester;

pub struct LoadTestRunner {
    memory_manager: Arc<MemoryManager>,
}

impl LoadTestRunner {
    pub async fn run_load_test(
        &self,
        config: LoadTestConfig,
    ) -> Result<LoadTestResult> {
        let mut tester = LoadTester::new(config.concurrent_users);

        // Add search tasks
        for _ in 0..config.search_requests {
            let manager = self.memory_manager.clone();
            tester.add_task(async move {
                manager.search("test query", &Filters::default(), 10).await
            });
        }

        // Add insert tasks
        for _ in 0..config.insert_requests {
            let manager = self.memory_manager.clone();
            tester.add_task(async move {
                manager
                    .create_memory("test content".to_string(), MemoryMetadata::default())
                    .await
            });
        }

        // Run test
        let results = tester.run(Duration::from_secs(config.duration_secs)).await?;

        Ok(LoadTestResult {
            total_requests: results.total_requests,
            successful_requests: results.successful_requests,
            failed_requests: results.failed_requests,
            average_latency: results.average_latency,
            p50_latency: results.p50_latency,
            p95_latency: results.p95_latency,
            p99_latency: results.p99_latency,
            requests_per_second: results.requests_per_second,
        })
    }
}

pub struct LoadTestConfig {
    pub concurrent_users: usize,
    pub search_requests: usize,
    pub insert_requests: usize,
    pub duration_secs: u64,
}

pub struct LoadTestResult {
    pub total_requests: usize,
    pub successful_requests: usize,
    pub failed_requests: usize,
    pub average_latency: Duration,
    pub p50_latency: Duration,
    pub p95_latency: Duration,
    pub p99_latency: Duration,
    pub requests_per_second: f64,
}

6. Dataset Management

6.1 Dataset Generation

pub struct DatasetGenerator {
    llm_client: Box<dyn LLMClient>,
}

impl DatasetGenerator {
    pub async fn generate_recall_dataset(
        &self,
        config: RecallDatasetConfig,
    ) -> Result<Dataset> {
        let mut test_cases = Vec::new();

        for i in 0..config.num_queries {
            // Generate query
            let query = self.generate_query(&config.domain).await?;

            // Generate relevant memories
            let relevant_memories = self.generate_relevant_memories(
                &query,
                config.num_relevant_per_query,
            ).await?;

            // Build expected results
            let expected_results = relevant_memories
                .iter()
                .enumerate()
                .map(|(j, memory)| ExpectedResult {
                    memory_id: memory.id.clone(),
                    relevance_score: 1.0 - (j as f64 * 0.1),
                    position: j,
                })
                .collect();

            test_cases.push(TestCase {
                id: format!("query_{}", i),
                query,
                expected_results,
                metadata: TestCaseMetadata {
                    domain: config.domain.clone(),
                    difficulty: config.difficulty,
                    ..Default::default()
                },
            });
        }

        Ok(Dataset {
            id: format!("recall_{}", Uuid::new_v4()),
            name: format!("Recall Dataset - {}", config.domain),
            description: format!("Generated recall dataset for {} domain", config.domain),
            test_cases,
            metadata: DatasetMetadata {
                dataset_type: DatasetType::Recall,
                created_at: Utc::now(),
                version: "1.0".to_string(),
            },
        })
    }

    async fn generate_query(&self, domain: &str) -> Result<String> {
        let prompt = format!(
            "Generate a natural language query for the {} domain. \
             The query should be specific and realistic.",
            domain
        );

        let query = self.llm_client.complete(&prompt).await?;
        Ok(query.trim().to_string())
    }

    async fn generate_relevant_memories(
        &self,
        query: &str,
        count: usize,
    ) -> Result<Vec<Memory>> {
        let prompt = format!(
            "Generate {} relevant memories for the following query:\n\n\
             Query: {}\n\n\
             Each memory should be a short, specific piece of information.\n\
             Format each memory on a new line.",
            count, query
        );

        let response = self.llm_client.complete(&prompt).await?;

        let memories: Vec<Memory> = response
            .lines()
            .filter(|line| !line.trim().is_empty())
            .take(count)
            .map(|content| Memory {
                id: Uuid::new_v4().to_string(),
                content: content.trim().to_string(),
                embedding: vec![0.0; 1536], // placeholder
                metadata: MemoryMetadata::default(),
                created_at: Utc::now(),
                updated_at: Utc::now(),
            })
            .collect();

        Ok(memories)
    }
}

pub struct RecallDatasetConfig {
    pub domain: String,
    pub num_queries: usize,
    pub num_relevant_per_query: usize,
    pub difficulty: Difficulty,
}

6.2 Dataset Loading

pub struct DatasetLoader;

impl DatasetLoader {
    pub fn load_from_file(path: &str) -> Result<Dataset> {
        let file = File::open(path)?;
        let reader = BufReader::new(file);

        let dataset: Dataset = serde_json::from_reader(reader)?;

        Ok(dataset)
    }

    pub fn save_to_file(dataset: &Dataset, path: &str) -> Result<()> {
        let file = File::create(path)?;
        let writer = BufWriter::new(file);

        serde_json::to_writer_pretty(writer, dataset)?;

        Ok(())
    }

    pub fn load_from_directory(dir: &str) -> Result<Vec<Dataset>> {
        let mut datasets = Vec::new();

        for entry in fs::read_dir(dir)? {
            let entry = entry?;
            let path = entry.path();

            if path.extension().and_then(|s| s.to_str()) == Some("json") {
                let dataset = Self::load_from_file(path.to_str().unwrap())?;
                datasets.push(dataset);
            }
        }

        Ok(datasets)
    }
}

7. Report Generation

7.1 Markdown Report

pub struct MarkdownReportGenerator;

impl MarkdownReportGenerator {
    pub fn generate_report(
        &self,
        result: &EvaluationResult,
    ) -> Result<String> {
        let mut report = String::new();

        // Title
        report.push_str(&format!("# {} Evaluation Report\n\n", result.evaluator_name));

        // Metadata
        report.push_str("## Metadata\n\n");
        report.push_str(&format!("- **Dataset ID**: {}\n", result.dataset_id));
        report.push_str(&format!("- **Timestamp**: {}\n", result.timestamp.format("%Y-%m-%d %H:%M:%S UTC")));
        report.push_str("\n");

        // Metrics
        report.push_str("## Metrics\n\n");
        report.push_str("| Metric | Value | Unit | Description |\n");
        report.push_str("|--------|-------|------|-------------|\n");

        for (name, metric) in &result.metrics {
            report.push_str(&format!(
                "| {} | {:.4} | {} | {} |\n",
                metric.name, metric.value, metric.unit, metric.description
            ));
        }

        report.push_str("\n");

        // Details
        report.push_str("## Details\n\n");
        report.push_str(&format!("- **Test Cases Passed**: {}/{}\n",
            result.details.test_cases_passed,
            result.details.test_cases_total
        ));

        if !result.details.errors.is_empty() {
            report.push_str("\n### Errors\n\n");
            for error in &result.details.errors {
                report.push_str(&format!("- **{}**: {}\n", error.test_case_id, error.message));
            }
        }

        if !result.details.warnings.is_empty() {
            report.push_str("\n### Warnings\n\n");
            for warning in &result.details.warnings {
                report.push_str(&format!("- **{}**: {}\n", warning.test_case_id, warning.message));
            }
        }

        Ok(report)
    }
}

7.2 JSON Report

pub struct JsonReportGenerator;

impl JsonReportGenerator {
    pub fn generate_report(
        &self,
        result: &EvaluationResult,
    ) -> Result<String> {
        serde_json::to_string_pretty(result)
            .map_err(|e| MemoryError::Serialization(e.to_string()))
    }
}

8. Usage Example

8.1 Running Evaluation

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize components
    let config = load_config("config.toml").await?;
    let manager = create_memory_manager(&config).await?;

    // Create dataset
    let dataset = generate_test_dataset(&manager).await?;

    // Create evaluator
    let evaluator = RecallEvaluator::new(
        RecallEvaluatorConfig {
            k_values: vec![5, 10, 20],
            similarity_thresholds: vec![0.7, 0.8, 0.9],
            max_results_per_query: 100,
        },
        manager.clone(),
    );

    // Run evaluation
    let result = evaluator.evaluate(&dataset).await?;

    // Generate report
    let report = MarkdownReportGenerator.generate_report(&result)?;

    // Save report
    fs::write("evaluation_report.md", report)?;

    println!("Evaluation completed successfully!");
    println!("Report saved to evaluation_report.md");

    Ok(())
}

8.2 Continuous Integration

# .github/workflows/evaluation.yml
name: Memory Evaluation

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  evaluate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Rust
        uses: actions-rs/toolchain@v1
        with:
          profile: release
          toolchain: stable

      - name: Run evaluation
        run: |
          cargo run --bin evaluation -- \
            --dataset ./datasets/test.json \
            --output ./reports/evaluation.md

      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: evaluation-report
          path: ./reports/evaluation.md

9. Summary

Cortex Memory's evaluation framework provides the following capabilities:

Comprehensive metrics: Covers recall, effectiveness, and performance dimensions
Flexible dataset management: Supports multiple data sources and generation methods
Multiple evaluators: Pluggable evaluator architecture
Rich report formats: Supports Markdown, JSON, and other formats
CI/CD integration: Easy to integrate into continuous integration workflows

This evaluation framework provides scientific, quantitative metrics for memory system optimization, ensuring continuous improvement of system quality and performance.

DEV Community: Sopaco

Architecture Docs Always Outdated? Let Code "Grow" Them Automatically

The Tech Lead's Pain: Documentation as Technical Debt

Knowledge Factory: Code as the Source of Documentation

Why Can It Stay "Always in Sync"?

Incremental Updates Instead of Full Regeneration

Freshness Scoring System

Dual-Track Output: Human-Readable, AI-Consumable

Source-First Trust Model

Four-Phase SDD: Making Design Reviewable

Quick Start

Who Is This For?

Knowledge Management in CI/CD: How Terrain Automates Document Updates

The Real Problem in CI/CD Pipelines

Core Capability: CLI-First, JSON Everywhere

Typical CI Usage: Auto-Refresh Knowledge on Merge

Environment Standardization: terrain env apply One-Click Deploy

Cross-Platform Distribution: npm + Pre-compiled Installers

Core Technology: Why Is It Pipeline-Friendly?

Native Rust Core, Runs Offline

Incremental Refresh Engine

Pipeline-Friendly Output Format

Complete CI Example

Who Is This For?

Struggling to Onboard New Projects? Terrain Gets You Up to Speed in 5 Minutes

The Pain We All Know

Core Scenario: Register → Initialize → Start Working

What Problems Does It Solve?

Core Technology: Why "Always in Sync"?

Incremental Update Mechanism

Three-Layer Retrieval Architecture

Native Rust Core: Works Offline

Quick Start

Who Is This For?

Your AI Coding Assistant Keeps Guessing Wrong? Terrain Puts It on Solid Ground

Sound Familiar?

Three Core Pillars

One Codebase, Two Audiences

Why Terrain?

Architecture at a Glance

Get Started

Community & Ecosystem

In the Age of AI Coding, What We Lack Isn't a Stronger Model — It's Ground for Agents to Stand On

A real pain point: Agents running "naked"

What Terrain is: an "engineering environment," not just another "AI tool"

Dual-track knowledge: one factory, two languages

A little background: not from scratch, but proven practice

Who is Terrain for?

How easy is it to get started?

Closing thoughts

Hermes Agent Codebase Packing Tool Usage Guide (repomix-rs High-Performance Edition)

What is repomix-rs?

🦀 Rust: The Infrastructure Choice for the AI Era

🔍 Why are developers choosing Rust?

🌟 Rust Replacement Cases: From Bun to Vite

Bun Case: The Rust Replacement for Node.js

Vite Case: Rust Rewriting of Frontend Build

Quick Start: No Installation Required, Just Run

Remote Repository Direct Packing

Fine-Grained Control: Which Files to Include, Which to Exclude

Output Format Selection

How to Connect to Hermes Agent (Critical Steps)

Running as an MCP Server (Recommended for Advanced Users)

Cursor / Claude Desktop Configuration

Claude Desktop (macOS)

Cursor

Performance Comparison: Original vs repomix-rs

Why repomix-rs Instead of the Original Repomix?

Frequently Asked Questions

📢 Quick Start with repomix-rs

repomix-rs: A Deep Dive into AI Code Context Infrastructure Built with Rus

Architecture Perspective — Examining repomix-rs's Design Philosophy, Crate Architecture, Data Lifecycle, and Relationship with the AI Agent Ecosystem from an Engineering Height

Table of Contents

1. Why Do We Need Code Context Infrastructure?

The LLM "Context Window Anxiety"

Token Economy

2. repomix-rs Architecture Overview

3. Crate Architecture Deep Dive

3.1 Cargo Workspace Design

3.2 Individual Crate Responsibilities

Environment Standardization: `terrain env apply` One-Click Deploy

`repomix-core` (Core Engine)

`repomix-config` (Configuration Schema)

`repomix-shared` (Cross-Crate Shared Types)

`repomix-cli` (CLI Layer)

`repomix-mcp` (MCP Server Layer)

Alignment with `.gitignore` Design