DEV Community: WonderLab

Codebase Knowledge Base (13): Evaluation — How to Know Your Knowledge Base Is Good Enough

WonderLab — Tue, 11 Aug 2026 02:31:03 +0000

Why Recall@5 Isn't Enough

Article 03 set a precedent: test the vector path on 30 retrieval questions, Recall@5 = 0.958. Every subsequent article with retrieval experiments used similar reporting.

Recall@5 is a reasonable starting metric — but it has three blind spots.

Blind spot 1: Precision. Recall@5 asks 'did the correct answer appear in the top 5 results?' But code retrieval often targets 'precisely locate that one function.' Appearing 5th versus 1st scores identically in Recall@5, but the practical difference is large — first place means direct hit, fifth place means manual scanning.

Blind spot 2: Task type. Article 08 established a three-path architecture, with different paths serving different tasks: vector for semantic exploration, graph for structural traversal, symbol for exact matching. Which path does a Recall@5 dataset evaluate? Combined evaluation of all three, or separate? Combined evaluation masks each path's specific weaknesses.

Blind spot 3: Impact analysis. One of the core values of a codebase knowledge base is 'if I change this function, what else does it affect?' The correct answer to this question is a set (all upstream callers), and evaluation must check not just 'did it return relevant results' but 'did it miss any important callers?' That's a recall problem — but not Recall@5.

Four-Dimensional Evaluation Metrics

Given codebase knowledge base characteristics, four purpose-built metrics:

Metric 1: Symbol Location Accuracy

Definition: For a set of 'where is this feature implemented' queries, the percentage where the correct answer (target function) appears in first place.

Why Top-1 instead of Top-5: Code retrieval 'success' means finding that specific function. Appearing in third place means manual filtering is still required. Top-1 accuracy measures direct-hit capability, which has the largest impact on actual engineering usability.

Evaluation dataset construction:

Sample queries:
Q1: "how to insert a new document into LightRAG"
A1: lightrag/lightrag.py::ainsert (function, line 1428)

Q2: "what query modes does LightRAG support"
A2: lightrag/base.py::QueryParam (class, line 83)

Q3: "where is document chunking strategy decided"
A3: lightrag/parser/routing.py::resolve_chunk_options

Q4: "what cleanup operations does deleting a document trigger"
A4: lightrag/lightrag.py::adelete_by_doc_id

Based on this series' experimental data: article 03's vector Recall@5 = 0.958, but Top-1 accuracy typically lands at 0.75-0.85 — the answer is in the top 5, but first-place rate is lower.

Metric 2: Semantic Search Recall

Definition: For a set of queries, the percentage where the correct answer appears in the top K results. K is typically 5 or 10.

Difference from Recall@5: The evaluation target isn't full-set retrieval — it specifically focuses on vocabulary mismatch scenarios: the user says 'file parsing,' the code calls it 'document ingestion pipeline'; the user says 'caching mechanism,' the code uses 'KV storage with TTL.'

These queries most clearly demonstrate the value of the vector path: BM25 fails at vocabulary mismatch, while vector embeddings bridge the lexical gap.

Key dataset construction principle: queries must deliberately use different terminology than the code. Otherwise BM25 can answer correctly, which doesn't test the vector path's contribution.

Strong test cases (vocabulary mismatch):
Q: "document deduplication logic" → A: compute_mdhash_id (operate.py)
Q: "LLM request rate control" → A: priority_limit_async_func_call (utils.py)
Q: "knowledge graph node merging" → A: _merge_nodes_then_upsert (operate.py)

Weak test cases (vocabulary directly matches — BM25 can do this too):
Q: "priority limit async func" → A: priority_limit_async_func_call

Metric 3: Impact Analysis Completeness

Definition: For a target function, given a query for 'all direct callers,' the intersection ratio between returned callers and actual callers.

This is the metric closest to real engineering value: before modifying a function, can the knowledge base tell you every affected location?

Formula:

Completeness = |returned callers ∩ actual callers| / |actual callers|

Real measurement from LightRAG (article 09): QueryParam has 19 actual callers, search_code("QueryParam") returned all 19. Completeness = 1.0 for this case.

Not all functions are this clean. BaseVectorStorage.upsert has fan_in = 268 — if the tool has a return limit (say limit=50), it would miss 218 callers and completeness would drop sharply. That's not a recall algorithm problem — it's a tool configuration problem. But evaluation surfaces that configuration gap.

Metric 4: History Query Hit Rate

Definition: For a set of 'why is this code written this way' queries, whether FILE_CHANGES_WITH edges or detect_changes historical data provide meaningful signals.

This is the hardest metric to quantify, since 'meaningful signal' is inherently subjective. In practice, binary scoring works:

1 = query results include directly relevant historical change information
0 = query results are empty or irrelevant

For priority_limit_async_func_call: query the historical change pattern of utils.py, get 'high-frequency change file' — hit, score 1.

For base.py::QueryParam: FILE_CHANGES_WITH shows base.py ↔ lightrag.py has strong coupling — hit, score 1.

Three Sources for Evaluation Datasets

When building evaluation datasets, codebase knowledge bases have a natural advantage over document RAG: the code itself is the annotation source.

Source 1: Extracted from test files

Test files are the best documentation of 'what this function should do.' For each test function, you can reverse-engineer the corresponding production retrieval question:

# tests/test_query.py
def test_hybrid_query():
    param = QueryParam(mode="hybrid")
    result = rag.query("what is LightRAG", param)
    ...

From this test, automatically generate:

Query: 'LightRAG hybrid query mode'
Expected answer: lightrag/base.py::QueryParam

LightRAG has over 200 test files — each is a potential evaluation data source.

Source 2: Extracted from git commit messages

Git commit messages frequently contain 'which function had what problem fixed' — reversible into queries:

commit: "fix: priority_limit_async_func_call deadlock when worker timeout"
→ Query: 'where is the concurrent LLM call timeout deadlock handled'
→ Expected answer: lightrag/utils.py::priority_limit_async_func_call

The characteristic of this dataset type: it tests 'real questions real engineers would ask,' which is more representative of actual usage than manually constructed questions.

Source 3: Manually constructed hard cases

The first two sources tend to cover 'parts of the codebase with explicit documentation.' But the scenarios where engineers most need the knowledge base are precisely those without documentation, where understanding requires reading the code. These need manual construction:

Hard case examples:
Q: "do SDK users and REST API users go through different code paths for document processing?"
A: Yes — SDK goes through ainsert (F-only), REST API goes through apipeline_enqueue_documents
   Finding the answer means locating the branch point between the two paths

Q: "changing a Kafka message format — which consumers are affected?" (cross-repo)
A: Requires cross-repo analysis, finding CROSS_ASYNC_CALLS edges

These cases don't need to be the majority of the evaluation set (around 20% is sufficient), but they're the dividing line between 'good enough' and 'genuinely useful.'

Reference Baselines for Four-Dimensional Metrics

Based on experimental data from the preceding twelve articles:

Metric	Weak baseline (vector only)	Three-path baseline	Four-path baseline
Symbol Location Accuracy (Top-1)	0.65-0.75	0.85-0.92	Similar to three-path (history path aids understanding, not location precision)
Semantic Search Recall (@5)	0.90-0.96	0.95-0.99	0.95-0.99
Impact Analysis Completeness	N/A (single path can't do impact analysis)	0.90-1.0 (depends on limit config)	0.90-1.0
History Query Hit Rate	0 (no history path)	0 (no history path)	0.70-0.85

How to read this table:

Semantic search recall is already high at the weak baseline (0.90+), because article 03 proved that AST-chunked vector retrieval is already quite strong. The three-path improvement shows up mainly in Top-1 accuracy and impact analysis — the two scenarios where vector retrieval alone performs poorly.

History Query Hit Rate is a four-path-only metric measuring whether the knowledge base can answer 'why' questions. The first three paths score 0 on it.

Wiring Evaluation into CI/CD

Evaluation isn't a one-time exercise — it's continuous monitoring. Code evolves and the knowledge base updates with it. After each incremental update, running a quick evaluation confirms that the update didn't degrade retrieval quality.

Minimal CI integration:

# Pseudocode — not runnable as-is
# Triggered in CI pipeline (after each PR merge)

def run_kb_quality_check():
    # Sample 50 questions from the annotation set
    # (random sample, covering all four metric types)
    questions = sample(EVAL_DATASET, n=50)

    results = {
        "symbol_accuracy": [],
        "semantic_recall": [],
        "impact_completeness": [],
        "history_hit": []
    }

    for q in questions:
        result = query_knowledge_base(q.query, path=q.path)
        results[q.metric_type].append(evaluate(result, q.expected))

    scores = {k: mean(v) for k, v in results.items()}

    # Alert thresholds
    THRESHOLDS = {
        "symbol_accuracy": 0.80,
        "semantic_recall": 0.90,
        "impact_completeness": 0.85,
        "history_hit": 0.65,
    }

    for metric, score in scores.items():
        if score < THRESHOLDS[metric]:
            alert(f"Knowledge base quality degraded: {metric} = {score:.2f}")

    return scores

This approach only requires maintaining an annotation dataset (50-200 questions is enough). CI runtime is roughly 5-15 minutes — acceptable overhead.

When to rebuild the annotation dataset: Large-scale refactors can invalidate old annotation sets if target functions were renamed or moved. Good practice is to record each function's git hash in the annotation set. If a target function no longer exists in the new version, automatically mark that question as 'stale' — it needs human update.

Series Retrospective: What Thirteen Articles Covered

A complete review at the final article.

This series began with one concrete question: how can engineers build a retrievable knowledge system for a large codebase?

Part 1 (Theory and Tools): Articles 01-02

Established two foundational understandings:

The core value of a codebase knowledge base isn't 'code generation' — it's 'code understanding': answering 'where is this,' 'what does it affect,' 'why is it this way'
Capability boundaries of existing tools: plain-text embedding, AST analysis, and knowledge graphs each have strengths, and each alone has blind spots

Part 2 (Core Technology): Articles 03-08

Conclusions driven by experimental data:

Article 03: AST function-level chunking Recall@5 = 0.958 — the ceiling for the vector-only path
Article 04: Four chunking strategies (F/R/V/P) — chunking granularity directly determines retrieval quality
Article 05: Graph and vector paths are not substitutes — Q8 (cross-file structural questions) proved the graph path is irreplaceable
Articles 06-07: Structural embeddings and hybrid search are marginal improvements; the core problem is 'routing to the wrong path,' not 'the path isn't strong enough'
Article 08: Three-path architecture — vector/graph/symbol as orthogonal paths, routed by query intent

Part 3 (Engineering Practice): Articles 09-13

From lab to production:

Article 09: codebase-memory-mcp three-path practice on a real project (LightRAG, 20,674 nodes)
Article 10: Incremental update three-tier decision tree — 80% of commits can be skipped at tier one
Article 11: Cross-repo analysis — 0 edges is a meaningful answer; LightRAG × graphrag = parallel alternatives
Article 12: Git history is the fourth path — 465 FILE_CHANGES_WITH edges reveal implicit coupling
Article 13: Evaluation framework — four-dimensional metrics, dataset construction, CI monitoring

The thread that runs through all thirteen:

This series has been making the same point throughout — code understanding is multi-dimensional, and no single signal covers all question types. The vector path is strong when vocabulary matches, the graph path is irreplaceable for structural traversal, the symbol path is unambiguous for precise location, and the history path is unique for understanding evolutionary context.

A good codebase knowledge base doesn't pick the strongest single path — it maps out each path's capability boundary clearly, then routes by question type.

Summary

The Codebase Knowledge Base series ends here.

Thirteen articles covered the full engineering lifecycle: from theory to tooling, from experiments to production, from single-repo to cross-repo, from building to maintaining.

If there's one sentence that captures the core conclusion of the whole series:

The quality of a codebase knowledge base depends not on how powerful the embedding model or how large the graph is — it depends on whether each path's capability boundaries are clearly understood, and whether the routing logic is correct.

Four retrieval paths, a three-tier decision tree, four-dimensional evaluation — none of this is complex engineering. It's what naturally emerges when you think clearly about the structure of the 'code understanding' problem.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #147: Ontology Playground — Microsoft's Zero-Backend Visual Ontology Learning Tool

WonderLab — Tue, 11 Aug 2026 02:29:42 +0000

Introduction

"Ontology is the map that lets AI understand what your business actually means. Building that map has always required fighting with dense standards and steep tooling."

This is article #147 in the "One Open Source Project a Day" series. Today's project is Ontology Playground — Microsoft's open-source zero-backend web tool for learning and designing ontologies, with a specific focus on Microsoft Fabric IQ.

Ontology appears constantly across knowledge engineering, semantic web, and enterprise AI domains, but it's notoriously hard to get started with. OWL, RDF, class hierarchies, property constraints — layers of standards and terminology that stay abstract without hands-on tooling. Ontology Playground converts those abstractions into clickable, draggable, live-preview visual graphs, paired with structured courses that go from fundamentals to working domain ontologies.

Zero backend. Zero server. Opens in any browser. 2,300 Stars. MIT license.

What You'll Learn

What ontology is and how it relates to knowledge graphs
What Microsoft Fabric IQ does, and the role ontology plays in it
Four core features: catalogue, designer, school, embeddable widget
How to export ontologies in RDF/OWL format for Fabric IQ
The embeddable widget: interactive ontology diagrams on any webpage with one line of HTML

Prerequisites

Basic knowledge graph concepts (nodes, edges, relationships)
Familiarity with data modeling thinking (entities, attributes, relationships)
No prior RDF/OWL experience needed — that's exactly what the Playground teaches

What Is Ontology

In philosophy, ontology studies "what exists." In computer science, an ontology is a formal description of knowledge in a domain: what types of entities exist, what properties they have, and what relationships connect them.

A retail domain ontology might describe:

Classes: Product, Order, Customer, Warehouse
Properties: Product has price (data property), Order placedBy Customer (object property)
Constraints: An Order must correspond to at least one Customer

Customer ──[placedBy]──→ Order ──[contains]──→ Product
   ↑                        ↑                      ↑
[Person]              [Transaction]          [StockItem]

This isn't a database schema. It captures business semantics: not just "these tables and columns exist," but "what these entities mean in business logic and what their relationships imply."

Ontology and AI

AI systems — especially LLM-driven agents — need more than data to understand business problems. They need semantic context:

"Order cancellation rate dropped 5%" requires the AI to know what an "order" is, what "cancellation" means as an operation, and 5% relative to what baseline
Without a semantic layer, the AI guesses or asks the user to re-explain in every conversation

That's the problem Microsoft Fabric IQ addresses.

Microsoft Fabric IQ: Semantic Foundation for Enterprise AI

Fabric IQ is the semantic layer in the Microsoft Fabric data platform. It links raw data lake content to business ontologies so AI agents can understand business meaning when querying data — not just execute SQL.

Without Fabric IQ:

AI Agent: "Show me high-value customer orders from the last 30 days"
→ AI needs to understand what "high-value customer" means
  (Is there a threshold? A customer tier table?)
→ Guesses or asks the user to explain

With Fabric IQ + ontology:

AI Agent: "Show me high-value customer orders from the last 30 days"
→ Queries the ontology: CustomerTier.HIGH_VALUE defined as annual spend > $10,000
→ Generates the correct query directly, no user explanation required

Ontology Playground is the tool for learning how to design and export ontologies for Fabric IQ.

Core Features

1. Ontology Catalogue

Six industry domains, pre-built and ready to explore:

Domain	Ontology	Entities	Relationships
Retail	Fourth Coffee	6	7
E-Commerce	Online Retail	5	6
Healthcare	Clinical System	5	6
Finance	Banking & Finance	5	6
Manufacturing	Industry 4.0	5	5
Education	University System	5	6

Each ontology has a shareable deep-link URL. Open one and you get an interactive graph — click nodes to inspect properties, follow edges to explore relationships.

2. Visual Designer

Drag-and-drop design interface:

Node operations: add classes, set names, add data properties (field + type)
Relationship operations: drag connections to create object properties, set relationship names and cardinality constraints
Live validation: structural errors shown in real time as you design
Undo/redo: 50 history levels
Five domain starter templates: avoid blank-canvas paralysis — pick the closest template and modify

Export to RDF/XML (.rdf or .owl) in exactly the format Microsoft Fabric IQ expects. Round-trip fidelity is verified by automated tests.

3. Ontology School: Nine Structured Courses

Fundamentals track (6 articles):

What ontology is and why enterprises need it
RDF and OWL standards introduction
Microsoft Fabric IQ overview
Ontology design patterns
From data model to ontology
Hands-on lab: 15-entity retail ontology from scratch

Seven domain learning paths: Four progressive articles each, from concepts to hands-on practice, with live embedded graphs in every article.

Interactive features:

Presentation/slides mode for every article (classroom-ready)
Multiple-choice quizzes with immediate feedback embedded throughout

4. Embeddable Widget

One line of HTML puts an interactive ontology graph on any webpage:

<!-- Embed the Healthcare ontology from the catalogue -->
<script
  src="https://microsoft.github.io/Ontology-Playground/widget.js"
  data-ontology-id="healthcare-clinical"
  data-theme="dark"
></script>

Widget options:

data-theme: dark or light
data-ontology-id: catalogue item ID
data-ontology-url: external RDF file URL
data-ontology-b64: base64-encoded inline RDF

Useful for embedding ontology diagrams into documentation, blog posts, or internal wikis.

Tech Stack

The entire application is pure static — zero server, zero backend:

Layer	Technology
Framework	React 19 + TypeScript 5
Graph rendering	Cytoscape.js (fcose layout)
State management	Zustand
Build tool	Vite
Animation	Framer Motion
Markdown	marked (build-time)

RDF handling: a custom parser and serializer with full round-trip fidelity for OWL classes, datatype properties, object properties, and cardinality constraints.

Deployment

GitHub Pages (Simplest)

# Fork the repository
# Go to Settings > Pages > enable GitHub Actions deployment
# Every push to main deploys automatically

Azure Static Web Apps

A GitHub Actions workflow is already configured in the repo. Connect your Azure SWA resource to the GitHub repository and it deploys on push.

Local Development

git clone https://github.com/microsoft/Ontology-Playground
cd Ontology-Playground
npm install
npm run dev
# Open http://localhost:5173

Optional: AI Ontology Builder

Enable via environment variable (requires Azure OpenAI):

VITE_ENABLE_AI_BUILDER=true

With this on, describe a business domain in natural language and the AI drafts an initial ontology structure. Then refine it in the visual designer.

One-Click Catalogue Contributions

GitHub device-flow sign-in unlocks a "submit to catalogue" flow: the app forks the repo, creates a branch, commits your RDF file and metadata, and opens a pull request — all automatically.

Resources

🌟 GitHub: microsoft/Ontology-Playground
🌐 Live Demo: microsoft.github.io/Ontology-Playground
📖 Microsoft Fabric IQ: Microsoft Fabric documentation

Summary

Ontology Playground solves a specific onboarding friction: ontology concepts aren't intrinsically hard, but the lack of immediately usable hands-on tooling has always been the real barrier. RDF/OWL specification documents are dense. Professional ontology editors like Protégé are complete but steep. Opening a browser, dragging out a retail ontology in five minutes, and exporting valid RDF — that difference matters significantly for learners.

The broader context: as AI agents deploy deeper into enterprise settings, "how do we make AI correctly understand business semantics" becomes a genuine engineering problem, not a theoretical one. Fabric IQ is Microsoft's architectural answer. Ontology Playground is the learning path to getting there.

If you're building on Microsoft Fabric for enterprise data, or if you're interested in semantic web and knowledge graphs but unsure where to start, this is worth an afternoon.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Open Source Project #146: Graphify — Turn Your Entire Codebase into a Queryable Knowledge Graph for AI Coding Assistants

WonderLab — Mon, 10 Aug 2026 08:23:39 +0000

Introduction

"AI coding assistants struggle to understand large codebases because they're grepping, not traversing. They find keywords, not relationships."

This is article #146 in the "One Open Source Project a Day" series. Today's project is Graphify — a Y Combinator-backed open-source tool that builds your entire project (code, docs, PDFs, images, video) into a queryable knowledge graph and serves it as a context layer for AI coding assistants.

Ask Claude Code or Cursor to explain how the auth module connects to the database, and you might get an accurate answer or you might get a plausible-sounding answer assembled from a handful of keyword-matched files that missed the critical middle layers. The root issue: AI assistants rely on keyword search and vector similarity to "understand" codebases. Neither approach captures actual structural relationships. Graphify replaces both: every function, class, module, and document becomes a node with explicit typed edges — and the AI traverses the graph to build context instead of guessing.

73,000 Stars in 2.5 months, 2.2M downloads, now at 101k Stars. Apache-2.0 + MIT.

What You'll Learn

The core difference between Graphify and RAG: graph traversal vs. vector similarity
tree-sitter AST local parsing: zero API calls for code analysis, nothing leaves your machine
Edge provenance: EXTRACTED / INFERRED / AMBIGUOUS tagging on every relationship
God Nodes: automatic high-impact node identification, blast radius visualization
Incremental updates: 3-file change patches in ~0.8 seconds, no full rebuild
One-command install and usage in Claude Code

Prerequisites

Experience with Claude Code, Cursor, or a similar AI coding tool
Basic familiarity with knowledge graph concepts (nodes, edges, relationships)
Basic Python environment comfort

Background: The "Project Blindness" Problem

A 500-file codebase. You ask Claude Code: "How does the auth module connect to the database?"

Its processing looks roughly like:

Search context for "auth"-related files
Pull several matching file contents into context
Generate an answer from those fragments

The problem is step 1. The search runs on keywords or vector similarity — not on the actual structural relationships in the code. It might:

Find three files containing the string "auth" but miss the critical middleware layer
Locate auth.py and db.py but not trace the call chain connecting them
Return a different answer if you ask the same question again

This isn't a model capability problem. It's a context construction problem.

Graphify's approach: before you ask anything, parse the entire codebase into a graph. Every function, class, module, and document is a node. Call relationships, import chains, and inheritance are typed directed edges. The AI assistant queries this graph to build context instead of grepping.

Core Architecture: Local AST + Optional LLM

Graphify splits project content into two categories and handles each differently:

Code files
    → tree-sitter AST local parsing
    → Zero API calls, nothing leaves the machine
    → Extracts function/class/module nodes + calls/imports/inherits edges

Docs / PDFs / images / video
    → Configured LLM backend (Anthropic / OpenAI / Gemini / Ollama / etc.)
    → Extracts semantic nodes and relationships

            ↓
    Unified knowledge graph

Output files:
├── graph.html       ← Interactive browser visualization
├── GRAPH_REPORT.md  ← Human-readable highlights and suggested questions
└── graph.json       ← Full queryable graph data

tree-sitter: Deterministic Code Parsing

tree-sitter is the industry-standard incremental AST parser used by Neovim, GitHub, and many other tools. Graphify uses it to parse code structure rather than asking an LLM to infer it.

The benefits:

Deterministic: same code always produces the same parse result
Zero API cost: code parsing is fully local, no LLM calls
Fast: AST parsing runs orders of magnitude faster than LLM inference
Private: code never leaves the machine

Supports 36+ programming languages: Python, TypeScript/JavaScript, Go, Rust, Java, C/C++, Ruby, C#, Kotlin, Swift, Scala, PHP, Lua, Zig, SQL, and more.

Edge Provenance

This is one of Graphify's most distinctive design decisions. Every edge in the graph carries a source tag:

Tag	Meaning
`EXTRACTED`	Derived directly from code structure, deterministic
`INFERRED`	Derived from LLM reasoning, some uncertainty
`AMBIGUOUS`	Origin unclear, worth human verification

Why it matters: when an AI assistant traverses graph paths to answer a question, it knows which relationships are "hard facts from the code" versus "model inferences." That directly affects how confidently the answer should be stated.

User asks: "How does the login function trigger a database write?"

AI traverses the graph and finds the path:
login() --[EXTRACTED: calls]--> validate_user()
validate_user() --[EXTRACTED: calls]--> db.query()
db.query() --[INFERRED: writes_to]--> users_table

The answer can explicitly note: the first two hops are code-confirmed;
the final hop is inferred.

God Nodes: Identifying High-Risk Points

Graphify automatically computes betweenness centrality across the graph to identify "God Nodes" — the files or functions that sit on the most paths between other nodes.

In a real codebase, God Nodes are typically:

A utility function imported by 20 different modules
The API layer file that bridges frontend and backend
A single service class holding all database connection logic

These nodes share one property: a bug in them has the widest blast radius. Graphify's visualization highlights God Nodes prominently. When reviewing AI-generated code changes, you see immediately whether the changed file is one of these high-centrality nodes.

Community Detection: Discovering Hidden Subsystem Boundaries

Graphify runs the Leiden algorithm on the graph to automatically cluster the codebase into functional subsystems — independent of directory structure.

Why this matters:

A utils/ folder might contain code that actually belongs to three different subsystems
Leiden clusters from actual call and import relationships, surfacing which files genuinely work together
Results appear in graph.html with distinct color coding per community

This is particularly useful for architecture understanding and refactoring planning in large codebases.

Incremental Updates: No Full Rebuild

A common pain with traditional RAG: when code changes, you re-embed the entire index. Hundreds of files might take minutes.

Graphify patches only the changed files, leaving every other node intact.

Official numbers: 500,000-node graph, 3 files changed, patch time: 0.8 seconds.

Query Interface

# Natural language query
graphify query "how does the login form connect to the users table?"
# → Returns the full path from UI through API layers to DB, with edge provenance tags

# Shortest path between two nodes
graphify path auth.login db.users

# Ask AI to explain a function using the graph as context
graphify explain src/auth/handler.py:validate_token

Supported Data Sources

Graphify ingests the full project, not just code:

Code (local AST, zero LLM)

36+ languages: Python, TS/JS, Go, Rust, Java, C/C++, Ruby, Kotlin, Swift, and more

Documents

Markdown, HTML, RST, YAML, TXT
.docx, .xlsx (optional extra)
PDF (optional extra)

Media

Images: PNG, JPG, WebP, GIF (vision extraction)
Video/audio: MP4, MOV, MP3, WAV (local transcription via faster-whisper)

Special formats

MCP configuration files
Package manifests: pyproject.toml, go.mod, pom.xml
Google Workspace: Docs, Sheets, Slides (via gws CLI)
YouTube URLs

Installation and Quick Start

Install

# Install CLI (uv recommended)
uv tool install graphifyy   # Note: PyPI package name is graphifyy (double y)

# Register skill with your AI assistant
graphify install

Use in Claude Code

# In Claude Code, run:
/graphify .              # Build graph for current directory

# Then ask project questions naturally —
# the AI traverses the graph instead of grepping

Output Files

graphify-out/
├── graph.html       ← Open in browser for interactive visualization
├── GRAPH_REPORT.md  ← Highlights, surprising connections, suggested questions
└── graph.json       ← Full graph data for programmatic queries

MCP Server Mode

# Start as MCP server
graphify mcp --transport stdio   # stdio transport
graphify mcp --transport http    # HTTP mode

Any MCP-compatible AI tool can then call into your code graph directly via tool calls.

Optional Graph Database Backends

# Push to Neo4j
graphify push --backend neo4j --uri bolt://localhost:7687

# FalkorDB
graphify push --backend falkordb

Benchmark Data

From the official BENCHMARKS.md, compared against popular memory/RAG systems:

Benchmark	Metric	Graphify	mem0	supermemory
LOCOMO (n=300)	recall@10	0.497	0.048	0.149
LOCOMO (n=300)	QA accuracy	45.3%	27.3%	49.7%
LongMemEval-S (n=50)	QA accuracy	76%	—	—
Graph build	LLM credits	0 (code)	per-token	per-token

On LongMemEval-S, Graphify ties dense RAG at 76% accuracy — with zero LLM cost for the code portion of the graph.

Supported AI Assistants

One command registers the skill across all:

Claude Code, Cursor, Codex, Gemini CLI, GitHub Copilot, Aider, Kilo Code, OpenCode, Factory Droid, Trae, Amp, Kiro, Devin CLI, and any other skill-compatible tool.

Resources

🌟 GitHub: Graphify-Labs/graphify
🌐 Website: graphify.com
📦 PyPI: graphifyy (pip install graphifyy)
🏢 Backed by: Y Combinator

Summary

Graphify's core insight: context quality determines answer quality more than model capability does.

The same Claude Sonnet given a handful of grep-matched file fragments versus a traced path through actual code relationships produces noticeably different answers in depth and accuracy. Graphify builds the latter: tree-sitter extracts precise code structure, Leiden identifies subsystem boundaries, betweenness centrality surfaces risk nodes, and all of it assembles into an auditable knowledge graph where every edge carries a provenance tag.

The edge provenance design (EXTRACTED vs INFERRED) deserves particular attention. In a large codebase, "is this relationship code-confirmed or model-inferred" is a question that changes what you do next. Being able to trace an AI answer back to graph paths — and knowing the confidence level of each hop — makes the difference between a useful tool and one you have to second-guess.

73,000 Stars in 2.5 months is a signal that this pain point is real. The grep-and-hope approach to codebase context has been the default long enough.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Codebase Knowledge Base (12): Git History Is the Fourth Retrieval Path

WonderLab — Mon, 10 Aug 2026 08:21:05 +0000

The Question That Three Paths Can't Answer

The previous articles demonstrated three-path retrieval repeatedly:

search_graph (vector path): ask 'which function handles document chunking strategy,' get QueryParam
trace_path (graph path): ask 'what is ainsert's full execution chain,' get 36 nodes
search_code (symbol path): ask 'where is BaseVectorStorage used,' get fan_in=268

But there's a class of question that all three paths fail to answer:

'Why is this code written this way?'

For example:

priority_limit_async_func_call is one of the highest-complexity functions in LightRAG (complexity=233, function body spanning 1,360 lines). Looking at the current code alone, it's hard to understand why a decorator that 'limits the number of concurrent async calls' needs to be this complex.

def priority_limit_async_func_call(
    max_size: int,
    llm_timeout: float = None,
    max_execution_timeout: float = None,
    max_task_duration: float = None,
    max_queue_size: int = 1000,
    cleanup_timeout: float = 2.0,
    queue_name: str = "limit_async",
    concurrency_group: str | None = None,
):
    """
    Enhanced priority-limited asynchronous function call decorator with robust timeout handling

    This decorator provides a comprehensive solution for managing concurrent LLM requests with:
    - Multi-layer timeout protection (LLM -> Worker -> Health Check -> User)
    - Task state tracking to prevent race conditions
    - Enhanced health check system with stuck task detection
    - Proper resource cleanup and error recovery
    - Optional cross-process global concurrency gating (gunicorn multi-worker)
    ...
    """

Why 'Multi-layer timeout protection'? Why 'Health check system with stuck task detection'? Why is there a concurrency_group parameter for multi-process scenarios?

trace_path can't answer these. Vector search can't find them either. The comments describe what the code does — not why it had to be done this way.

The answer is in git history.

The Nature of the Fourth Question Type

Three-path retrieval covers the current state of the code:

Path	Question answered	Data source
Vector	What does this do, where is this concept	Semantic embeddings of current code
Graph	Who calls it, what is the full execution chain	AST + call graph of current code
Symbol	Precise location, full blast radius	Symbol index of current code
History	Why is it written this way, when was it added, how many times has it changed	Git commit history

The history path isn't a supplement to the other three — it's a completely different information dimension. The current state of the code is just one cross-section of its historical evolution; to understand that cross-section, sometimes you have to see how it got there.

codebase-memory-mcp encodes git history into queryable knowledge from two angles:

FILE_CHANGES_WITH edges: mined from git log — 'which files are frequently modified together' — encoded as explicit relationships in the graph
detect_changes history patterns: using the since parameter to look back across any time window, tracking change frequency for any file or module

FILE_CHANGES_WITH: Turning Git History into Graph Edges

LightRAG's knowledge graph has 465 FILE_CHANGES_WITH edges.

These edges don't come from static code analysis — no Python import or function call could produce them. They're mined from git commit history: if file A and file B frequently appear in the same commit historically, a bidirectional A ↔ B edge is created in the graph.

Reading through those 465 edges, three clear patterns emerge.

Pattern 1: Documentation and code evolve in sync

FileProcessingPipeline.md ↔ routing.py
FileProcessingPipeline.md ↔ parser.py
FileProcessingPipeline.md ↔ param_schema.py
FileProcessingPipeline.md ↔ test_hint_params.py

LightRAGSidecarFormat-zh.md ↔ pipeline.py
ParagraphSemanticChunking.md ↔ paragraph_semantic.py
ParagraphSemanticChunking.md ↔ test_paragraph_semantic_table_split.py
MilvusConfigurationGuide.md ↔ milvus_impl.py

This reveals that LightRAG's maintainers have a strong habit: documentation and code are updated simultaneously. When routing.py changes, FileProcessingPipeline.md almost always changes too. When paragraph_semantic.py gets a new feature, the corresponding documentation and test files update at the same time.

For a codebase knowledge base, this means: when you index a change to routing.py, FileProcessingPipeline.md also needs to be in scope — not because they have a call relationship, but because historical evidence says they always move together.

Pattern 2: Implementation and test coupling

backfill.py ↔ test_backfill.py
backfill.py ↔ test_sidecar_backfill_integration.py
_markdown.py ↔ test_markdown.py
anthropic.py ↔ test_anthropic_client_cleanup.py

This is a test coverage health signal: high co-change frequency between implementation files and test files means these modules have had tests keeping pace throughout their evolution.

The inverse is equally informative: if an implementation file has no FILE_CHANGES_WITH edges pointing to test files, that's worth noting.

Pattern 3: Cross-module architectural coupling

base.py ↔ lightrag.py
addon_params.py ↔ pipeline.py
_vision_utils.py ↔ pipeline.py
azure_openai.py ↔ openai.py
anthropic.py ↔ lmdeploy.py ↔ hf.py ↔ lollms.py

The most interesting case here is the LLM adapter cluster: anthropic.py, lmdeploy.py, hf.py, and lollms.py have FILE_CHANGES_WITH edges connecting them to each other.

From static analysis, these files have no import relationships and no direct call relationships — they're parallel LLM backend adapters, each implementing the same interface. But historically, modifying one almost always means modifying the others at the same time.

This reveals an important architectural convention: when the LLM interface spec changes, all adapters must update in sync. That rule isn't written in any comment. It only left traces in git history.

Using detect_changes to Trace History Windows

Beyond FILE_CHANGES_WITH edges, the since parameter of detect_changes can look back across any time window of change patterns.

In the incremental update article (article 10), we used detect_changes(since="HEAD~5") for a live demo — 5 commits containing only README and CI changes, zero code changes, the correct answer being 'do nothing.'

But since has another use case: understanding a module's evolution history.

For example, running a historical window analysis on utils.py (where priority_limit_async_func_call lives):

detect_changes(
    project="LightRAG",
    since="HEAD~50",   # look at the last 50 commits
    scope="lightrag/utils.py"
)

If the result shows utils.py appeared in 15 of the last 50 commits — while the project has around 200 files — then utils.py's change frequency is roughly 15 times the average.

This is exactly what explains why priority_limit_async_func_call grew to 1,360 lines with complexity 233: it wasn't written this way in one sitting. It was repeatedly repaired and hardened under production load — every new timeout edge case, every new concurrency bug, every new multi-process scenario added another layer of protection.

The 'Multi-layer timeout protection,' 'stuck task detection,' and 'cross-process concurrency gating' in the docstring are all scars from real production incidents.

Four Uses for the History Path in Practice

Use 1: Understanding 'why so complex'

When you encounter a high-complexity function, the history path distinguishes two very different situations:

Case A: The function grew from 100 to 1,000 lines in a single commit — that's a large refactor or feature expansion, and there's usually a complete commit message explaining why
Case B: The function slowly ballooned across dozens of commits, gaining a few dozen lines each time — it's been continuously patching production issues, and each chunk of new code is a hotfix

priority_limit_async_func_call is Case B: 1,360 lines of function body, reflecting every concurrency problem LightRAG has encountered under high-load LLM calls.

Use 2: Finding 'implicit coupling' relationships

Static analysis can surface call relationships (A calls B), inheritance (A extends B), and import relationships (A imports B). But there's one type of relationship it can't extract from current code: A and B must be modified in sync, or the system breaks.

The LLM adapter cluster is the canonical example: anthropic.py and openai.py don't call each other, but if you change the interface spec and update only openai.py while forgetting anthropic.py, something will break.

FILE_CHANGES_WITH edges make this 'implicit coupling' into a queryable graph relationship.

Use 3: Assessing change risk

When a function is being modified, its historical change frequency is a proxy for risk:

High-frequency changes = this area has been historically unstable, modifications are likely to introduce bugs
Low-frequency changes = this area has been stable, changes are relatively lower risk
Never changed = either extremely stable foundational code, or long-neglected dead code

Combining complexity (current state) with historical change frequency builds a two-dimensional risk matrix:

                  Low complexity     High complexity
High change freq  [Watch carefully]  [High risk, full analysis required]
Low change freq   [Relatively safe]  [May be stable complex logic]

Use 4: Detecting documentation-code sync gaps

The docs↔code FILE_CHANGES_WITH edges effectively define which documentation should stay in sync with which code.

MilvusConfigurationGuide.md ↔ milvus_impl.py means: every time milvus_impl.py has an interface change, MilvusConfigurationGuide.md should update too. If a diff changes milvus_impl.py without touching the corresponding documentation — that's an automatically detectable problem.

The Complete Four-Path Picture

With the history path added, the full codebase knowledge base looks like:

Question type                         Path           Core tool
────────────────────────────────────────────────────────────────────
'Where is this feature implemented?'  Vector         search_graph(query=...)
'Who calls it, what's the blast rad?' Graph          trace_path(mode=calls)
'Precise location, all callers'       Symbol         search_code(pattern)
'Why written this way, when added'    History        FILE_CHANGES_WITH + detect_changes

These four paths aren't competing — they're complementary dimensions. An engineer trying to understand unfamiliar code typically needs all four running in parallel: the vector path to find the entry point ('what is this'), the graph path to trace the structure ('how does it work'), the symbol path to confirm scope ('what does changing it touch'), and the history path to understand the context ('why is it this way').

Summary

Git history isn't an optional add-on for a codebase knowledge base — it's the fourth dimension of code understanding, without which the picture is incomplete.

codebase-memory-mcp encodes historical knowledge into the graph two ways:

FILE_CHANGES_WITH edges: mined from commit history, making 'these files always change together' into queryable graph relationships
detect_changes historical lookback: scanning any time window via the since parameter, quantifying change frequency, explaining why some areas are more complex than others

LightRAG's 465 FILE_CHANGES_WITH edges reveal three patterns: docs↔code synchronization conventions, impl↔test coverage signals, and cross-adapter interface contracts. Each pattern answers a class of 'why' questions — the kind of answer that static code analysis can never provide.

Next article (the final one): we step back and ask how to evaluate whether a codebase knowledge base is 'good enough' — what metrics matter, how to design evaluation datasets, and what production systems should track when Recall@5 is no longer the only number that counts.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Codebase Knowledge Base (11): Cross-Repo Scenarios — When One Service Calls Another

WonderLab — Sun, 09 Aug 2026 03:12:27 +0000

An Experiment That Returned Zero

Running cross-repo-intelligence across LightRAG and graphrag:

index_repository(
    repo_path="/path/to/LightRAG",
    mode="cross-repo-intelligence",
    target_projects=["mnt-hdd-...-graphrag"]
)

Output:

status: success
cross_http_calls: 0
cross_async_calls: 0
cross_channel: 0
cross_grpc_calls: 0
total_cross_edges: 0
elapsed_ms: 109

Zero cross-repo edges.

The first instinct is 'the tool found nothing — something failed.' That's wrong. This result is entirely correct, and it's highly informative.

LightRAG and graphrag are two open-source frameworks that do similar things — both graph-based RAG — but they are parallel alternatives, not an integrated system. No LightRAG code calls graphrag's API, and vice versa. cross-repo-intelligence finds integration relationships, not functional similarity. There was never an integration relationship between them; returning 0 is the right answer.

This matters because it draws a clear boundary: cross-repo analysis solves a completely different class of problem than single-repo analysis.

Single-Repo vs Cross-Repo: Two Different Problems

Looking back at the first ten articles, all the questions a single-repo knowledge base can answer live within one repository boundary:

'Where is this function?' (symbol path)
'Who calls it?' (graph path, inbound)
'What does it call?' (graph path, outbound)
'Which functions frequently change together?' (FILE_CHANGES_WITH)

These answers are all in the same codebase. The call graph is complete. BFS from any node works.

But when system scope expands across multiple repositories, a class of questions appears that single-repo analysis physically cannot answer:

'Which interfaces of Service B does Service A call?'
'If I modify Service B's /query endpoint, which upstream services are affected?'
'Service C changed the message format it sends to Service D via message queue — can D detect this?'
'In the entire microservice network, who are the central nodes, who are the islands?'

These answers span repository boundaries. A single-repo call graph stops dead at the service boundary — the graph becomes a map with 'dangling terminal nodes' that call some external URL, but don't know who handles that URL.

cross-repo-intelligence connects those dangling nodes.

How Cross-Repo Edges Are Discovered

The core logic of cross-repo-intelligence mode: match Route nodes against HTTP_CALLS nodes across indexed projects.

Concretely:

Discover server-side routes: Scan all indexed projects for Route nodes (HTTP endpoint definitions). Build a path → project routing table.
Discover client-side calls: Scan HTTP_CALLS edges (HTTP requests in the code). Extract the target URLs or path patterns.
Path matching: For each HTTP_CALLS edge, look for a matching Route in other projects. When found, create a CROSS_HTTP_CALLS edge that crosses the repository boundary, connecting the caller to the callee.

The same logic applies to:

CROSS_ASYNC_CALLS: message queues (Kafka topic names, RabbitMQ exchanges, etc.)
CROSS_GRPC_CALLS: gRPC service/method names
CROSS_CHANNEL: other named channels (WebSocket channels, Redis pub/sub keys)

This explains why LightRAG × graphrag returns 0: no LightRAG code sends HTTP requests to graphrag's routes, and graphrag doesn't call LightRAG's API. The tool didn't fail — it correctly reported that there is no integration relationship between these two systems.

What Each System's Route Profile Actually Looks Like

Even without cross-repo edges, examining each system's routes reveals their roles immediately.

LightRAG: REST API server

Route nodes (lightrag/api/routers/):
  GET  /query
  POST /query
  GET  /query/stream
  POST /query/stream
  GET  /health
  POST /login
  GET  /auth-status
  GET  /graph/label/list
  POST /documents/paginated
  ...

LightRAG exposes a full HTTP service interface — document management, querying, graph browsing, health checks. From the cross-repo perspective, it's a callee: if another service sends requests to http://lightrag-host/query, that request becomes one end of a cross-repo edge.

The corresponding code structure: create_query_routes and create_document_routes are massive route registration functions (1,566 lines, complexity 118, cognitive complexity 266) — among the hardest functions in the project, because they handle all the HTTP boundary cases.

graphrag: LLM client

Route nodes (graphrag/):
  PATCH  /            (API root — to LiteLLM)
  external: https://litellm.ai
  external: https://raw.githubusercontent.com/.../cspell.schema.json

graphrag isn't a service — it's a command-line tool + Python SDK. Its 'routes' are HTTP calls it makes to external LLM services (LiteLLM) and GitHub, not endpoints it exposes. From the cross-repo perspective, it's a pure caller — if you deploy both graphrag and LightRAG in the same system, graphrag might call LiteLLM (creating CROSS_HTTP_CALLS edges), but graphrag won't call LightRAG and LightRAG won't call graphrag.

This role difference is immediately visible from the route profile: one has real REST API routes, the other only has outbound HTTP client calls.

Where Cross-Repo Analysis Has Real Engineering Value

If LightRAG × graphrag has no cross-repo edges, what kind of system does?

A typical microservice architecture:

Frontend App
  └─ CROSS_HTTP_CALLS → API Gateway (gateway-service)
                              └─ CROSS_HTTP_CALLS → User Service
                              └─ CROSS_HTTP_CALLS → Auth Service
                              └─ CROSS_HTTP_CALLS → Order Service
                                    └─ CROSS_ASYNC_CALLS → Kafka: order.created
                                                                └─ CROSS_ASYNC_CALLS → Notification Service

In this architecture, if User Service's GET /users/{id} interface is being modified (say, a returned field is being removed), cross-repo analysis tells you which services have CROSS_HTTP_CALLS edges pointing to that route — and need to be updated in sync. Pure single-repo analysis can't answer this at all.

cross_service mode trace_path:

trace_path(
    function_name="get_user_by_id",
    project="user-service",
    mode="cross_service",
    depth=3
)

This call follows HTTP_CALLS → CROSS_HTTP_CALLS → Route edges across service boundaries, returning the full cross-service call chain — from some frontend handler all the way down to a database query in User Service.

Three Practical Cross-Repo Use Cases

Use case 1: Service dependency map

After indexing all microservice projects, query the cross-repo call graph:

MATCH (a)-[r:CROSS_HTTP_CALLS]->(b)
RETURN a.file_path, r.path, b.file_path

The result is the entire system's service dependency map. Which services are 'hubs' (called by many others), which are 'islands' (no cross-repo calls), whether there are circular dependencies — one query surfaces all of it.

Use case 2: Interface change impact assessment

# 1. Find the interface definition
search_code("GET /api/v2/payments", project="payment-service")

# 2. Find cross-repo callers
trace_path("get_payment", mode="cross_service", direction="inbound")
→ returns all upstream services with CROSS_HTTP_CALLS edges to this route

Before modifying /api/v2/payments, know which services depend on it. Without cross-repo analysis, this requires manually grep-ing that URL across every repository in the organization — a process with frequent misses.

Use case 3: Message contract tracing

If services communicate via Kafka:

MATCH (producer)-[r:CROSS_ASYNC_CALLS]->(consumer)
WHERE r.channel = "order.completed"
RETURN producer.file_path, consumer.file_path

Find all producers and consumers of the order.completed message. When the message schema needs to change, the full blast radius is visible immediately.

Zero Cross-Repo Edges Still Has Value: Interface Design Comparison

Back to LightRAG and graphrag — no cross-repo edges, but both indexes exist. Holding two indexed projects simultaneously enables something useful even without integration edges: interface design comparison.

LightRAG's query interface:

# LightRAG
async def aquery(self, query: str, param: QueryParam) -> str

graphrag's query interface:

# graphrag
async def local_search(
    config: GraphRagConfig,
    entities: pd.DataFrame,
    communities: pd.DataFrame,
    community_reports: pd.DataFrame,
    text_units: pd.DataFrame,
    relationships: pd.DataFrame,
    covariates: pd.DataFrame | None,
    community_level: int,
    response_type: str,
    query: str,
) -> tuple[str | dict, str | list[pd.DataFrame] | dict[str, pd.DataFrame]]

Two frameworks doing the same category of thing, with radically different interface philosophies. LightRAG encapsulates all configuration in QueryParam (designed for a service runtime). graphrag expects the caller to manage every DataFrame directly (designed for a batch analytics pipeline).

These interface designs reflect completely different target use cases — and this kind of cross-project interface comparison is one of the most valuable analyses for technology selection decisions. It's also a side-effect of having both projects indexed at the same time, without requiring any cross-repo edges.

Summary

Cross-repo analysis isn't 'single-repo analysis made bigger.' It solves a genuinely new class of problem: connecting knowledge at service boundaries.

Three core conclusions:

Zero cross-repo edges is a meaningful answer: it says two systems have no integration relationship — not that the analysis failed. Don't conflate 'the tool found something' with 'the tool worked correctly.'
Cross-repo analysis works by route matching: it matches Route nodes exposed by one service against HTTP_CALLS / ASYNC_CALLS edges from another. That's what it can find, and only that. Functional similarity, code style comparison, duplicate logic detection — those are different tools' jobs.
Projects with no cross-repo edges still benefit from joint indexing: a cross-repo knowledge base lets you query multiple projects simultaneously. Even without call edges, parallel indexes have analytical value — like interface design comparison.

Next article: we turn to knowledge base evaluation — how do you know your knowledge base is 'good enough'? What metrics matter, how to design evaluation datasets, and what production systems should track when Recall@5 is no longer the only metric that matters.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #145: Open Code Review — Alibaba's Battle-Tested AI Code Review Tool, 1/9 the Tokens of a General Agent

WonderLab — Sun, 09 Aug 2026 03:10:24 +0000

Introduction

"The problem with general-purpose agents doing code review isn't that they miss bugs. It's that they don't know which files to read, they drift on line numbers, and they burn 9x the tokens doing it."

This is article #145 in the "One Open Source Project a Day" series. Today's project is Open Code Review — Alibaba's open-sourcing of its internal AI code review tool, released in 2026 after years of serving tens of thousands of engineers internally and identifying "millions of code defects."

Tools that use LLMs for code review are common, but most just dump the diff into a model and wait for output. Open Code Review starts from a different place: a hybrid architecture where deterministic engineering pipelines handle the things that must not be wrong (file selection, line positioning, rule matching), and the LLM agent handles only what genuinely requires dynamic judgment. The result: same underlying model, higher precision and F1, roughly one-ninth the token consumption of a general-purpose agent.

18,100 Stars. 1,200 Forks. Apache 2.0.

What You'll Learn

The hybrid architecture design rationale: why separate "deterministic" from "agent"
ocr review (incremental diff review) vs ocr scan (full-file audit)
Delegation mode: your existing agent runs the review; no OCR API key needed
Benchmark numbers comparing against Claude Code as a general agent
GitHub Actions integration and three SLA review levels
Defect patterns specific to AI-generated code

Prerequisites

Familiarity with Git workflows (branch, diff, PR)
Basic understanding of CI/CD concepts
Experience with Claude Code or similar AI coding tools is helpful

Background

From Internal Tooling to Open Source

Open Code Review wasn't purpose-built for open-source release — it's Alibaba's production-running internal system published externally. That distinction matters: its design decisions come from real large-scale operational experience, not from first-principles speculation.

Before open-sourcing, the internal system:

Served tens of thousands of engineers
Identified millions of code defects
Handled the genuine complexity of production-scale codebases

The Problem with General Agents for Code Review

Sending a diff directly to Claude Code or GPT for code review carries systematic weaknesses:

Problem	How It Appears
Incomplete file coverage	Agent autonomously picks which files to read; misses critical cross-file dependencies
Position drift	Line numbers in comments land on incorrect lines
Inconsistent quality	Same diff reviewed twice produces different severity assessments
Token waste	General agent reads large amounts of unnecessary context

The root cause: general agents hand all decisions to the LLM for flexibility, including decisions that deterministic code could handle precisely.

Core Architecture: Hybrid Design

Open Code Review's design philosophy: let deterministic things be handled deterministically; let the agent handle only what genuinely requires dynamic judgment.

Git changes
    ↓
[Deterministic Layer]
  ├── Precise file selection (no misses, no extras)
  ├── Smart bundle grouping (related files → isolated sub-agent contexts)
  ├── Template engine rule matching (NPE, thread safety, XSS, SQL injection)
  └── External positioning + reflection modules (accurate line-level placement)
    ↓
[Agent Layer]
  ├── Scenario-tuned prompts and toolsets
  ├── Optimized from analysis of production tool-call traces
  └── Dynamic context retrieval and tool calls
    ↓
Code review output (precise line-level comments)

Deterministic layer handles:

Selecting exactly the right files from git changes — no gaps, no noise
Grouping related files into Bundles, each processed in an isolated sub-agent context (divide and conquer)
Matching common defect rules via template engine rather than asking LLM to reason from scratch each time
Ensuring comment line numbers are accurate with no position drift

Agent layer handles:

Prompts and toolsets tuned from analyzing production tool-call traces at scale
Complex cross-file reasoning that genuinely needs inference
Dynamic tool calls

The result: higher precision (the deterministic layer removes LLM randomness from structure), lower token usage (the agent processes only the genuinely hard parts).

Benchmark Data

The Open Code Review team built a benchmark from:

50 open-source repositories
200 PRs
10 programming languages
1,505 annotated issues (hand-labeled by 80+ engineers)

Compared against Claude Code (general agent mode) using the same underlying model:

Metric	Open Code Review	Claude Code (general agent)
Precision	Higher	Lower
F1	Higher	Lower
Recall	Lower (deliberate)	Higher
Token usage	~1/9	Baseline

On the recall tradeoff: Open Code Review deliberately designs for lower recall than a general agent. This isn't a limitation — it's a choice. Fewer false alarms and higher precision are more valuable than comprehensive coverage in a CI/CD gate. Noise that blocks PRs creates review fatigue faster than any genuine bug.

What the token difference means in practice: For 100 PRs, Open Code Review consumes roughly 11% of what a general agent would. In CI/CD where every PR triggers a review, this determines whether the tool is financially viable. Nine-to-one cost difference changes the calculation entirely.

Two Review Modes

`ocr review`: Incremental Diff Review (Most Common)

Reviews git diffs — staged changes, unstaged changes, branch ranges, or single commits:

# Review current working tree changes (staged + unstaged)
ocr review

# Review a branch range
ocr review --from main --to feature/new-auth

# Review a single commit
ocr review --commit abc1234

# Output formats
ocr review --output json
ocr review --output sarif   # importable into GitHub Security

`ocr scan`: Full-File Audit

No git history required — reviews file content directly:

# Audit a directory
ocr scan --path internal/

# Audit a single file
ocr scan --path src/auth/handler.go

# Generate HTML report
ocr scan --path src/ --output html

Good fit for: security audits on inherited codebases, quality assessment of legacy code with no git history, reviewing code snippets without a repository context.

Delegation Mode

One of Open Code Review's most interesting design choices.

Standard mode: OCR's agent layer uses an LLM API you configure (Anthropic/OpenAI key required) to run the review.

Delegation mode: OCR handles only the deterministic layer (file selection, bundle grouping, rule resolution), then delegates the review task to your existing agent (Claude Code, Codex, Cursor, etc.), using that agent's own LLM.

# Preview the delegation plan (see how OCR intends to split the task)
ocr delegate preview

# Generate rule descriptions for specific files for the agent to review
ocr delegate rule src/main.go src/handler.go

Why it's useful:

You already have a Claude Code subscription or API key — no separate OCR key needed
OCR's deterministic layer handles file selection and rule resolution; your agent focuses on understanding and judgment
Integrates into your existing agent workflow without switching tools

GitHub Actions Integration

Thirty-second setup. Every PR triggers a review automatically:

name: AI Code Review
on: [pull_request]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: raye-deng/open-code-review@v1
        with:
          sla: L2          # Review depth: L1 / L2 / L3
          threshold: 60    # Fail if quality score drops below 60
          github-token: ${{ secrets.GITHUB_TOKEN }}

Three SLA Levels

Level	Description	Best For
L1	Fast structural detection, no AI required	Quick PRs, low-risk changes
L2	Adds semantic analysis and embedding	Standard feature development
L3	Full LLM deep scan: cross-file coherence, logic bug detection, confidence scoring	Core paths, security-sensitive code

AI-Generated Code Defect Detection

Open Code Review positions itself on GitHub Marketplace as "the first open-source CI/CD quality gate built specifically for AI-generated code" — detecting defect patterns common in LLM output that traditional linters miss entirely:

Defect Type	What It Looks Like
Hallucinated imports	Packages that don't exist (verified live against npm/PyPI/Maven)
Stale APIs	Deprecated methods present in training data but removed
Context window artifacts	Logic contradictions spanning multiple files
Over-engineering	Unnecessary abstractions and dead code
Security anti-patterns	Hardcoded secrets, `eval()` usage

These problems appear at significantly higher rates in AI-generated code than in human-written code — standard linters aren't looking for them.

Supported languages: TypeScript/JavaScript, Python, Java, Go, Kotlin (6 languages).

Installation and Configuration

Install

# npm (recommended)
npm install -g @alibaba-group/open-code-review

# Requires Git >= 2.41
git --version

Configure the LLM

ocr config provider
# Interactive setup: choose Anthropic / OpenAI / custom compatible endpoint

Fine-grained configuration via .ocrrc.yml:

sla: L3
ai:
  embedding:
    provider: ollama
    model: nomic-embed-text
  llm:
    provider: ollama       # local Ollama supported
    model: qwen3-coder     # any OpenAI-compatible model

First Run

cd my-project

# Review current changes
ocr review

# List sessions (resume support built in)
ocr session list

Resources

🌟 GitHub: alibaba/open-code-review
🌐 Docs: open-codereview.ai
🏪 GitHub Marketplace: Open Code Review Action
📦 npm: @alibaba-group/open-code-review

Summary

Open Code Review makes one thing clear: in the specific domain of code review, using engineering constraints around the LLM produces better results than letting the LLM operate freely.

The problem with general agents for code review isn't insufficient LLM capability — it's that handing all decisions to the LLM introduces unnecessary randomness and token waste where deterministic code could be precise. Handling file selection, line positioning, and rule matching with deterministic pipelines focuses the LLM's attention on the parts that actually need inference.

This design principle is worth generalizing: not "how do we make the AI do better" but "which parts should the AI never have been doing." It's a conclusion that most AI tooling teams arrive at after hitting the same walls at scale. Alibaba packaged the lessons from that experience and released them along with the code.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Codebase Knowledge Base (10): Incremental Updates — When to Rebuild the Index, and Which Parts

WonderLab — Sat, 08 Aug 2026 11:12:47 +0000

A Problem That's Easy to Overlook

Build the codebase knowledge base, run the first query, watch vector search and graph traversal return exactly what you wanted — and it's tempting to call the job done.

But the code keeps moving. New PRs merge. Functions get renamed. Interfaces change. New files appear, old modules get deleted. Three months later, the function signatures in your index may be stale, the call graph might point to a function that has been moved, and the embeddings might still represent a rewritten implementation. Your queries return plausible but wrong answers — and the failure is silent.

Knowledge base freshness is a harder problem than building the knowledge base in the first place, and it's much easier to ignore.

Full rebuild is simple: code changed, delete the index and rerun everything. For a medium-scale codebase that might take hours — you can't do it on every commit. The opposite extreme is 'never update' — let the index drift permanently away from the actual code, and watch retrieval quality degrade quietly over months.

This article charts the practical middle path: precisely determine which changes affect what, and rebuild only the parts that genuinely need it.

Starting From a Real detect_changes Output

Running detect_changes on the LightRAG codebase (HEAD~5 to HEAD) returns:

changed_files: [
  ".github/workflows/copilot-setup-steps.yml",
  ".github/workflows/tests.yml",
  "lightrag_webui/bun.lock",
  "lightrag_webui/package.json",
  "lightrag_webui/README.md",
  "README-ja.md",
  "README.md",
  "README-zh.md"
]
changed_count: 8
impacted_symbols: [
  {name: "jobs", label: "Variable", file: ".github/workflows/..."},
  {name: "LightRAG WebUI", label: "Section", file: "lightrag_webui/README.md"},
  {name: "Installation", label: "Section", file: "lightrag_webui/README.md"},
  ...
]

Eight changed files. The impacted_symbols list contains only CI variables (jobs, on) and Markdown headings (Section) — zero Functions, zero Methods, zero Classes.

The correct response to this change set is: do nothing.

Not because the changes are unimportant (updating CI config and README is real work) — but because none of these 8 files have any content that would affect the knowledge base's three retrieval paths. There's nothing from these files in the vector index, no nodes from them in the call graph, nothing to find in the symbol index. A full rebuild would be pure waste: hours of compute time and API costs, for zero improvement in retrieval quality.

This brings us to the first tier.

Tier 1: Does This Change Affect Retrievable Content?

Different file types have very different impacts on the knowledge base:

File type	Affects vector index	Affects call graph	Affects symbol index	Decision
`.py` function changes	Yes	Yes	Yes	Incremental update needed
`.ts` / `.tsx` changes	Yes (if indexed)	Yes	Yes	Incremental update needed
Test files (`test_*.py`)	Optional	Policy-dependent	Optional	Configure per team
CI config (`.yml`)	No	No	No	Skip
Markdown / README	No	No	No	Skip
`package.json` / lockfiles	No	No	No	Skip
Config files (`.env`, `.toml`)	No	No	No	Skip

The decision rule: Does impacted_symbols contain any entries with label equal to Function, Method, or Class? If not, skip this update entirely.

In practice, encode this as a git hook or CI step:

# Pseudocode — not runnable as-is
def should_update_index(changed_files: list[str]) -> bool:
    CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.js'}
    return any(
        Path(f).suffix in CODE_EXTENSIONS
        for f in changed_files
    )

For LightRAG's 5-commit window: {'.yml', '.lock', '.json', '.md'} — no code files. Return False. Index update cost for this batch = 0.

Tier 2: Which Functions Changed — Rebuild Only Those

When Tier 1 determines there are real code changes, enter Tier 2.

Suppose a change touches lightrag/operate.py and lightrag/pipeline.py. detect_changes returns the specific symbols that changed in those files. The strategy is:

Re-embed only the changed functions. Leave everything else untouched.

The underlying assumption: the embedding unit is a function, and functions are relatively self-contained semantic units. When a function's implementation changes, only its position in vector space needs to update — every other function's embedding vector remains valid.

The same principle applies to call graph updates:

If naive_query in operate.py adds a new call to _find_related_text_unit_from_entities, update only naive_query's outgoing edges — don't rebuild the whole graph.
If a function is deleted, remove the corresponding node and all its edges.

Incremental update cost scales as (changed function count / total function count). LightRAG has 7,761 Function nodes. A change affecting 50 functions costs roughly 0.6% of a full rebuild.

Tier 3: High-Impact Changes — Trace the Call Chain

Tiers 1 and 2 handle 'I know which functions directly changed.' Tier 3 answers a deeper question: do these direct changes affect functions that themselves haven't changed?

Two signals matter here.

3a. Call graph propagation

Suppose a low-level function in parse_document.py changes its interface. It's called by analyze_multimodal in pipeline.py, which is called by the top-level API in lightrag.py. Every function in this chain may have changed behavior in the new version — even though only the leaf function is in the diff.

trace_path("changed_function", mode=calls, direction=inbound)
→ returns all upstream callers that depend on this function
→ flag these callers as 'potentially affected' (re-analyze, don't necessarily re-embed)

3b. FILE_CHANGES_WITH: the hidden signal of temporal coupling

This is an edge type in codebase-memory-mcp that's easy to overlook but highly valuable. Its meaning: in historical git commits, these two files were frequently modified together.

Looking at the LightRAG knowledge graph's FILE_CHANGES_WITH edges reveals a clear pattern:

Selected FILE_CHANGES_WITH pairs:
  FileProcessingPipeline.md ↔ routing.py      # docs and routing code evolve together
  FileProcessingPipeline.md ↔ parser.py       # docs and parser evolve together
  operate.py ↔ utils.py                       # extraction logic and utilities coupled
  operate.py ↔ prompt.py                      # extraction logic and prompts move together
  pipeline.py ↔ utils_pipeline.py             # pipeline core and pipeline utilities
  base.py ↔ lightrag.py                       # abstract base and main class evolve together
  config.py ↔ lightrag.py                     # configuration and main class
  chunk_schema.py ↔ pipeline.py               # chunk schema and pipeline
  document_routes.py ↔ routing.py             # document API and routing layer

These relationships weren't derived from static analysis — they were mined from git history. What they tell you:

When operate.py changes, history shows a high probability that utils.py and prompt.py changed simultaneously — even if this specific diff doesn't show them changing. If you only rebuild operate.py's index, you might miss a synchronized adjustment in prompt.py's knowledge extraction logic.

This is coupling that static analysis can never surface. Function A doesn't call function B. File X doesn't import file Y. But historical evidence says they always move together — that's an implicit architectural convention, only visible in temporal data.

Practical use: when computing an incremental update, include the FILE_CHANGES_WITH neighbors of changed files in your inspection scope, even if they didn't change in this specific commit.

The Three-Tier Decision Tree

Assembled into an operational flow:

Tier 1: detect_changes(since=last_index_commit)
  ├── impacted_symbols all Section/Variable/Module?
  │   └── Skip. Check again next time.
  └── Function/Method/Class changes present?
      │
Tier 2: Extract list of changed functions
  ├── Re-embed changed functions (overwrite old vectors)
  ├── Update call graph edges locally (add/remove/modify)
  └── Update symbol index (renames/deletes/additions)
      │
Tier 3: Expand impact scope
  ├── trace_path(changed_fn, direction=inbound)
  │   └── Flag upstream callers as 'potentially affected'
  └── FILE_CHANGES_WITH(changed_files)
      └── Add historical co-change neighbors to next review cycle

Cost estimates for LightRAG:

Scenario	Changed functions	Update cost	% of full rebuild
CI + README update	0	0	0%
Small feature (1-2 files)	5-20	Minutes	< 1%
Module refactor (5-10 files)	50-200	10-30 min	2-4%
Large refactor (20+ files)	500+	Trigger full rebuild	100%

A Real Data Point: High-Complexity Functions and Change Cost

Querying for the highest-complexity functions in LightRAG:

MATCH (f) WHERE f.label IN ['Function','Method'] AND f.complexity > 15
RETURN f.name, f.file_path, f.complexity, f.transitive_loop_depth
ORDER BY f.complexity DESC LIMIT 5

Results (excluding the bundled swagger-ui file):

adelete_by_doc_id      lightrag/lightrag.py      complexity=131  transitive_loop_depth=14
create_document_routes lightrag/api/routers/...   complexity=118
analyze_multimodal     lightrag/pipeline.py       complexity=116  transitive_loop_depth=14
create_app             lightrag/api/lightrag_server.py  complexity=91
openai_complete_if_cache lightrag/llm/openai.py   complexity=89

adelete_by_doc_id has complexity=131 and transitive_loop_depth=14 — its execution path nests 14 levels deep, making it one of the hardest functions in the codebase to reason about and one of the most likely to propagate change effects.

If this function appears in a change set, Tier 3 isn't optional — every change to it requires a full inbound call-chain trace to verify that upstream callers' behavioral assumptions still hold.

Conversely, if the change is a paragraph in README.md, this complexity data is irrelevant to you. The first value of detect_changes is that it filters out the noisy commits — CI tweaks, doc fixes, dependency bumps — at Tier 1, before any analysis needs to happen.

Summary

Incremental updates are fundamentally a precision problem, not an engineering-effort problem. Full rebuild is simpler to implement. Incremental updates require finer-grained decision logic. But once that logic exists, the payoff compounds.

A three-tier incremental update system lets roughly 80% of commits (docs, config, test-only) bypass index updates entirely, and the remaining 20% (real code changes) rebuild only the affected fraction. For a continuously evolving codebase, this isn't an optimization — it's what makes the knowledge base viable over months and years.

Next article: we extend the scope from a single codebase to multi-repo scenarios — when one service calls another service's API, or when microservices communicate through message queues, how should the knowledge base build connections that span repository boundaries?

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #144: Omnigent — Databricks' Meta-Harness for Unified Control of Claude Code, Codex, and Cursor

WonderLab — Sat, 08 Aug 2026 11:03:43 +0000

Introduction

"You use Claude Code for planning and Codex for fast execution. But you're manually copying output between them."

This is article #144 in the "One Open Source Project a Day" series. Today's project is Omnigent — an AI agent meta-orchestration framework open-sourced by Databricks co-founder Matei Zaharia in June 2026. 8,100 Stars.

Omnigent's position is specific: it's not another AI coding agent. It's the layer that sits above existing agents — a meta-harness. The analogy Databricks makes: Kubernetes doesn't replace servers, it adds orchestration, policy, and observability on top of them. Omnigent doesn't replace Claude Code; it makes Claude Code, Codex, Cursor, and Pi manageable components of one unified system instead of a pile of independent tools.

If you've used multiple AI coding tools, you've probably hit this: each tool has its own interface, its own API key management, its own context and session, and configuring team-level permissions and budgets for each separately is its own overhead. Omnigent targets exactly that layer.

8,100 Stars. 1,200 Forks. Apache 2.0. Alpha stage.

What You'll Learn

What a "meta-harness" is and how Omnigent relates to agent frameworks
Policy Governance: three-tier policy stacking for token budgets and tool access
Cloud sandboxes: local OS isolation (bwrap/seatbelt) plus cloud execution (Modal/E2B/Daytona)
Multi-agent orchestration: YAML sub-agents, parallel worktrees, cross-vendor reviewers
Real-time collaboration: session sharing, co-driving, forking
MLflow Tracing integration: unified observability across harnesses

Prerequisites

Experience with at least one of Claude Code, Codex, or Cursor
Basic understanding of AI coding agents (context, tool calls, system prompts)
Familiarity with YAML configuration format

Background: The Fragmented Agent Landscape

The 2026 AI coding tool landscape: Claude Code with claude-sdk and claude-native modes, Codex with its own CLI, Cursor embedded in the IDE, OpenCode, Hermes, Pi — each with distinct capabilities, each a separate island.

Matei Zaharia (inventor of Apache Spark, Databricks co-founder) ran into this while driving AI agent adoption across Databricks' 5,000+ engineer organization:

Different agents have different strengths; one task often needs multiple agents working together
Agent output requires manual transfer between tools
Team-level governance has no infrastructure (who accesses what tools? what's the token budget?)
No unified observability — no visibility into which agent spent what, or what decisions it made

Omnigent distills that deployment experience into a framework that tackles the whole problem.

Core Architecture: The Meta-Harness Layer

Omnigent's architecture has three layers:

┌─────────────────────────────────────────┐
│           Omnigent Server               │
│   Policy governance / collaboration /   │
│   observability / API                   │
├─────────────────────────────────────────┤
│           Omnigent Runner               │
│   Sandboxing / unified API /            │
│   session management                    │
├──────────┬──────────┬───────────────────┤
│ claude-sdk│  codex  │  cursor / pi / …  │
│ (harness) │(harness) │   (harnesses)    │
└──────────┴──────────┴───────────────────┘

Harness (execution layer): the existing agent tools — claude-sdk, claude-native, codex, codex-native, cursor, hermes, opencode, pi, openai-agents
Runner (runtime layer): wraps any harness in a sandboxed session with a uniform API
Server (service layer): policy enforcement, session sharing, cross-device sync

The Kubernetes analogy holds: Kubernetes doesn't replace servers — it adds orchestration, elasticity, and policy above them. Omnigent doesn't replace Claude Code — it adds unified control above it.

Policy Governance

Policies are one of Omnigent's core differentiators. Three tiers stack:

Server-wide (shared across all agents)
    ↓
Agent-level (defined per agent)
    ↓
Session-level (dynamic adjustments per task)

Cost Control

# Token budget in the agent definition
policies:
  cost:
    max_tokens_per_session: 50000
    max_tokens_per_turn: 5000
    alert_at: 40000   # alert when approaching limit
    pause_at: 50000   # pause and wait for human approval

Agents that exceed budget pause automatically, waiting for a human to decide whether to continue. This solves the common problem of agents quietly burning large token budgets on complex tasks with no warning.

Tool Access Control

Policies support stateful conditional restrictions — not just static allow/block lists:

policies:
  tools:
    # After downloading an npm package, require approval before git push
    - condition: "after:tool:npm_install"
      require_approval: ["git_push"]

    # Never allow the agent to read .env files directly
    - block: ["read_file:.env*"]

    # Agent can call APIs, but credentials injected via proxy — agent never sees them
    - network:
        egress_proxy: "https://my-credential-proxy"
        direct_credential_access: false

Stateful rules like "require git push approval after npm install" were effectively impossible with per-tool agent configurations.

Cloud Sandboxes

Omnigent supports two sandboxing approaches:

Local OS Sandboxing

Linux: bwrap (Bubblewrap) namespace isolation
macOS: seatbelt sandbox
Windows: Windows Job Objects (limited support; no PTY/tmux wrapper)

Cloud Execution Environments

Route agent execution to cloud sandboxes instead of local machines:

Provider	Best for
Modal	Stateless function-style tasks
E2B	Code execution sandbox
Daytona	Development environments
Kubernetes	Enterprise private cloud
CoreWeave	GPU-intensive tasks
Databricks	Data analytics tasks

Multi-Agent Orchestration: YAML-Defined

Omnigent agents are defined in YAML. An agent can call other agents as tools:

# Basic agent definition
name: my_coder
prompt: |
  You are an expert Python developer.
  Focus on correctness and test coverage.
executor:
  harness: claude-sdk
tools:
  run_tests:
    type: function
    callable: mypackage.testing.run_pytest
  search_docs:
    type: mcp
    url: https://docs-mcp.example.com

Multi-Agent Pattern: Polly

Omnigent ships a built-in example called Polly — a supervisor orchestrator demonstrating the parallel worktree pattern:

name: polly
prompt: |
  You are a supervisor coordinating multiple coding agents.
  Delegate coding tasks to sub-agents, then route diffs to reviewers.
tools:
  coder_a:
    type: agent
    harness: claude-sdk
    prompt: "Implement the feature in a git worktree"
  coder_b:
    type: agent
    harness: codex
    prompt: "Implement the same feature independently"
  reviewer_anthropic:
    type: agent
    harness: claude-sdk
    prompt: "Review the diff for correctness"
  reviewer_openai:
    type: agent
    harness: codex
    prompt: "Review the diff for security issues"

The pattern:

Two coders on different harnesses implement the same feature in parallel git worktrees
Diffs get routed to two reviewers from different vendors
The supervisor synthesizes review feedback and makes the final call

Dual-Headed Agent: Debby

The other bundled example is Debby — runs Claude and GPT side-by-side with a /debate mode:

User question
    → Claude produces answer A
    → GPT produces answer B
    → /debate triggers: each model critiques the other's answer
    → Debby synthesizes both arguments into a final response

Real-Time Collaboration

Omnigent turns agent sessions into multi-person workspaces:

Session Sharing

# Generate a shareable URL
omnigent share <session_id>
# → https://omnigent.ai/s/abc123

Teammates open the URL, see the agent's output in real time, inject commands, and take over the session.

Session Forking

# Fork from an existing session — same context, independent development
omnigent run --fork <session_id>

Useful in code review: the main agent finishes an implementation, fork into two sessions with different reviewer agents, merge the conclusions back into the main session.

Authentication

OIDC login: Google, GitHub, Okta, Microsoft
Invite-only accounts via single-use invite links

MLflow Tracing Integration

The Omnigent + MLflow integration provides unified observability across all harnesses. Configuration is minimal:

uv tool install omnigent mlflow

export MLFLOW_TRACKING_URI="http://your-mlflow-server:5000"
export OMNIGENT_TELEMETRY_ENABLED="true"

omnigent run

Every agent execution automatically logs to MLflow:

Each conversation turn (prompt + response)
Tool invocations with arguments, results, and timing
Per-turn token consumption
Session metadata (model name, agent name, harness type)

Practical uses:

Model comparison: run the same task with Claude vs. GPT, compare cost and quality
A/B testing: evaluate different MCP server providers on cost/performance
Team insights: which harness excels at planning vs. execution? Which request types run slow? How much time goes to features vs. bugs?

Installation and Quick Start

# Install (uv recommended)
uv tool install omnigent

# Or pip
pip install omnigent

# Start (picks a model, opens localhost:6767 automatically)
omnigent

# Short alias
omni

# Upgrade
omni upgrade

The browser opens http://localhost:6767 with a GUI for configuring agents and viewing sessions. Terminal interaction also works directly.

Deploy to a Server

# Docker
docker run -p 6767:6767 omnigentai/omnigent

# Supported platforms: Render, Railway, Fly.io, Cloudflare

Server deployment gives teams a shared Omnigent instance with OIDC login — all policies and collaboration features managed centrally.

Resources

🌟 GitHub: omnigent-ai/omnigent
🌐 Website: omnigent.ai
📝 Databricks blog: "Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents"
📊 MLflow integration: mlflow.org/blog/omnigent-mlflow-tracing
👤 Lead author: Matei Zaharia (Databricks co-founder, inventor of Apache Spark)

Summary

Omnigent doesn't try to build a better AI coding agent. It addresses what happens when you already have multiple agents and need to turn them into a coherent system.

Databricks' experience deploying AI agents across 5,000+ engineers surfaced the problem clearly: tools fragment, policy has no infrastructure, team collaboration has no shared foundation, observability is nearly absent. Omnigent packages the answer — meta-harness layer + policy governance + sandboxing + collaboration + observability — into one operational framework for running agents at scale.

The "meta-harness" concept itself is worth holding onto. As AI coding tools multiply and specialize, the need for this layer only grows. Today it's Claude Code and Codex. Tomorrow it might be five more specialized tools. The problems Omnigent solves — policy, cost, collaboration, observability — appear in every combination.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Codebase Knowledge Base (09): codebase-memory-mcp in the Wild — Three-Path Retrieval on LightRAG

WonderLab — Fri, 07 Aug 2026 05:29:27 +0000

Eight Articles Built the Ship. Now Let's Sail It.

If you've followed the series so far, your working memory should hold a clear checklist:

Article 03: AST function-level chunking + vector retrieval, Recall@5 = 0.958 — the ceiling for text-only approaches
Article 05: Graph retrieval rescues Q8, but BFS noise breaks Q1 — it's a one-for-one trade
Articles 06 & 07: Structure-aware embedding and hybrid search can't actually bridge the gap
Article 08: Three orthogonal paths — vector for semantics, graph for structure, symbol for exact matches — route by query intent

All of that was measured on a 30-function toy codebase. The toy was honest. But it was still a toy.

Today we change the venue.

LightRAG is one of the highest-starred open-source knowledge graph RAG frameworks on GitHub. The codebase has been fully indexed by codebase-memory-mcp: 20,674 nodes, 94,517 edges, covering 409 Python files and 101 TypeScript files (Web UI). That's a medium-scale real project — large enough to make everything from the earlier experiments matter.

This article's job is straightforward: take three real questions, fire each one down the appropriate retrieval path, and spread the results flat so you can see exactly what each path delivers.

A Quick Map Before We Start

Before firing queries, one orientation pass.

LightRAG's name might imply a simple retrieval utility, but the actual code structure is considerably richer. Running get_architecture reveals the most important signal first: layer boundaries.

entry layer:   kg/, chunker/        ← storage adapters, chunkers
core layer:    base, api, parser    ← abstract base classes, REST API, parsers
internal:      examples/, tests/   ← examples, tests (not exported)

This immediately tells you: if you want to understand 'how does LightRAG process a document,' the main arena is lightrag/, the entry point is lightrag.py, and storage implementations live in kg/ (10+ backends: Neo4j, MongoDB, Qdrant, Milvus, PostgreSQL, ...).

Now for the retrieval runs.

Path 1: Vector — Ask in Natural Language, Get a Concept Entry Point

Question: What retrieval modes does LightRAG support, and when should each one be used?

This is a classic capability query: the caller doesn't know what the relevant function or class is called — they just know what they want to understand. This is exactly where the vector path excels.

Tool: search_graph (BM25 + vector dual index)

query: "query search hybrid retrieval mode"
label: Method
limit: 5

Mode	Description
`local`	Local context retrieval, focuses on the entity neighborhood around the query
`global`	Global knowledge retrieval, aggregates relationships across documents
`hybrid`	local + global combined
`naive`	Plain vector search, no knowledge graph
`mix`	Knowledge graph + vector fusion (the default)
`bypass`	Pass directly to LLM, skip retrieval entirely

Path 2: Graph — Follow the Call Chain, Understand What a Function Triggers

Question: What does ainsert() call? What is the full execution path for document ingestion?

This is a structural query: the caller knows the entry point name and wants to understand the complete execution path downstream. The graph path owns this category.

Tool: trace_path (call graph BFS)

function_name: ainsert
mode: calls
direction: outbound
depth: 2

Returns (hop=1 direct callees, hop=2 indirect):

hop=1 direct:
  apipeline_enqueue_documents   (pipeline.py)
  apipeline_process_enqueue_documents   (pipeline.py)
  generate_track_id   (utils.py)
  resolve_chunk_options   (parser/routing.py)

hop=2 indirect (inside pipeline):
  _run_pipeline_batch
  _validate_and_fix_document_consistency
  _atomic_release_busy_or_consume_pending
  compute_mdhash_id
  sanitize_text_for_encoding
  normalize_document_file_path
  filter_keys        (BaseKVStorage)
  upsert             (BaseVectorStorage)
  get_by_id          (BaseVectorStorage)
  get_docs_by_statuses  (DocStatusStorage)
  get_namespace_data
  get_namespace_lock
  ...  (36 nodes total)

These 36 nodes form the complete document ingestion path. But the node list alone isn't the full story — the source code is:

async def ainsert(
    self,
    input: str | list[str],
    split_by_character: str | None = None,
    ...
) -> str:
    """Async insert documents with checkpoint support (fixed-token chunking only).

    SDK convenience entry point. It **always** chunks with the fixed-token
    (F) strategy: ``process_options`` is intentionally not passed, so the
    document runs the F chunker. ...

    The LightRAG **server / REST API does not call this method** — it
    ingests via :meth:`apipeline_enqueue_documents` +
    :meth:`apipeline_process_enqueue_documents` with a per-document
    ``process_options`` selector, which is how F/R/V/P are chosen there.
    """
    chunk_opts = resolve_chunk_options(
        self.addon_params,
        split_by_character=split_by_character,
        split_by_character_only=split_by_character_only,
    )
    await self.apipeline_enqueue_documents(input, ids, file_paths, track_id, chunk_options=chunk_opts)
    await self.apipeline_process_enqueue_documents()
    return track_id

There is a critical design decision buried in this source that you would never see from the function name alone:

ainsert only supports the fixed-token chunking strategy (F strategy). If you want recursive-character (R), semantic-vector (V), or paragraph-semantic (P) chunking, you cannot call ainsert — you must call apipeline_enqueue_documents + apipeline_process_enqueue_documents directly with an explicit process_options.

Equally: LightRAG's REST API server doesn't call ainsert either — it goes directly to the pipeline layer. So SDK users and REST API users are actually running different code paths.

This is what the graph path uniquely delivers: not just 'here is a function,' but 'here is what role this function plays in the system.' A vector search would probably return ainsert for 'how to insert documents.' But it wouldn't tell you that this method is intentionally designed as a simplified F-only entry point, and what that means for how you should actually use it.

Path 3: Symbol — Precise Hit, Zero Ambiguity

Question: Which parts of the codebase call BaseVectorStorage.upsert?

This is an impact analysis query: the caller wants to do a refactor or security audit and needs to know every caller of a given interface. No semantic understanding is required — only precise matching.

Tool: search_code (graph-augmented grep)

pattern: "BaseVectorStorage"
mode: compact
limit: 5

Results show upsert (the core method of BaseVectorStorage) has fan_in = 268 — one of the most heavily called methods in the entire project.

A more targeted query:

pattern: "QueryParam"

Immediately locates:

QueryParam   lightrag/base.py:83    (Class, in_degree=19)
  ↑ called by 19 locations:
  - lightrag/lightrag.py:2056  (method: query)
  - lightrag/lightrag.py:2091  (method: aquery)
  - lightrag/operate.py:4323   (_perform_kg_search)
  - lightrag/lightrag.py:2344  (aquery_llm)
  ...

19 callers, all precisely located with file and line number. If you're modifying QueryParam's interface — deprecating a mode, adding a parameter — this list is your blast radius assessment.

The symbol path is graph-augmented grep. Plain grep returns where a string appears. The symbol path also tells you each hit's position in the call graph — whether it's an entry point, who calls it, how high its in-degree is. That's enough to answer 'how big a change is this?' in seconds.

All Three in Combination: One Real Engineering Task, Three Perspectives

The three examples above demonstrated each path independently. Now, a scenario closer to real engineering:

Task: Understand LightRAG's full document ingestion flow, in order to modify the chunking strategy

This is a question an engineer would ask before making a feature change. The answer isn't in any single function — it needs multiple perspectives assembled into a map.

Step 1: Vector path — locate the concept entry points

search_graph("document chunking strategy pipeline insert")
→ ainsert (lightrag.py:1428)
→ ainsert_custom_chunks (lightrag.py — deprecated)
→ resolve_chunk_options (parser/routing.py)

The vector path reveals 'which entry points in this system relate to chunking,' and along the way flags that ainsert_custom_chunks is deprecated.

Step 2: Graph path — trace the execution chain

trace_path("ainsert", mode=calls, depth=2)
→ hop=1: apipeline_enqueue_documents → resolve_chunk_options
→ hop=2: _run_pipeline_batch → [various Storage.upsert]

The graph path shows: chunking strategy selection happens in resolve_chunk_options (before enqueue), actual chunking happens in _run_pipeline_batch (pipeline execution), and the result gets written to multiple storage backends. If you want to change chunking strategy, modify parser/routing.py — not ainsert in lightrag.py.

Step 3: Symbol path — pin the interface definition

search_code("resolve_chunk_options")
→ lightrag/parser/routing.py:chunk_strategy_key
→ lightrag/parser/routing.py:slim_chunk_options
→ lightrag/parser/routing.py:default_chunker_config

The symbol path pins three related functions in routing.py, with their exact reference locations in the ainsert call chain.

After three steps you know: change chunking strategy in parser/routing.py; ainsert via SDK only supports F strategy, all others require calling the pipeline layer directly; storage writes are abstracted through BaseVectorStorage.upsert, so chunking changes won't affect storage backends. That's a complete engineering change map — each path contributed exactly what it's best at.

The Counter-Intuitive Finding: Scale Makes the Graph Path More Valuable

On the toy codebase (30 functions), trace_path expanded to a handful of nodes — the graph path's value wasn't obvious. On LightRAG (7,761 functions + 3,569 methods), the same call trace expanded to 36 nodes covering the complete path from user API to storage abstraction.

This isn't linear growth — it's a scale amplification effect: the larger the codebase, the richer any function's neighborhood, the harder it becomes for pure vector retrieval to surface structural relationships, and the greater the relative advantage of the graph path.

This is the Q8 problem from Article 05 reflected in a real codebase. ainsert and _run_pipeline_batch are semantically distant — 'high-level document insertion API' vs 'low-level batch pipeline executor' — no embedding model would bring those two close. But the call graph connects them in one hop.

When the question is 'what would change if I modify this?' or 'who calls this function?' — those questions have the same answer structure in a 5,000-function codebase as in a 30-function one: the answer is only in the graph.

Summary

Nine articles in, this series has tracked a single thread from first principles to production tooling: measure the ceiling of each single-path approach, identify where each fails, then design around the failure modes.

A final statement of where each path belongs:

Vector path: You don't know what the function is called, just what it does — search_graph(query=...) is the first move
Graph path: You know the entry point, want to understand its execution chain and blast radius — trace_path is the workhorse
Symbol path: You know the exact function or class name, want its definition and all references — search_code returns results in under a second

These aren't competing approaches. They're orthogonal dimensions. Any task that requires 'understand an unfamiliar codebase' needs all three to build a complete cognitive map.

The central conclusion, one more time:

Codebase semantic understanding is multi-dimensional. Every single signal has a blind spot. Pick the right tool, keep the paths distinct — that's what engineering-quality retrieval actually looks like.

For more details on codebase-memory-mcp and its full feature set, visit the GitHub repository.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #143: Better Harness — A Five-Dimension Workflow Evaluator for AI Coding Agents That Reviews the Loop, Not the Diff

WonderLab — Fri, 07 Aug 2026 05:26:15 +0000

Introduction

"Your AI coding agent generates code fast. Your workflow is the bottleneck."

This is article #143 in the "One Open Source Project a Day" series. Today's project is Better Harness — QoderAI's open-source evaluation tool that analyzes the workflow surrounding an AI coding agent, not just the code it produces.

Most evaluations of AI coding agents focus on output quality: test pass rates, defect density, functional correctness. Better Harness takes a different position: agents fail not because the model is weak, but because the surrounding workflow has gaps. Fuzzy goals, no reusable execution paths, unvalidated changes, bypassed quality checks, lessons that evaporate after each session — these problems don't show up in diffs. They only appear when you audit the workflow itself.

Better Harness collects project and session evidence, evaluates workflow health across five dimensions, and outputs prioritized findings — each with a scoped, actionable repair plan.

1,500 Stars. MIT license. Supports Claude Code, Codex, GitHub Copilot, Cursor, and Qwen Code.

What You'll Learn

Better Harness's core model: feedforward guides and feedback sensors
What each of the five dimensions evaluates and what evidence counts
The architecture: three independent evidence agents running in parallel
Report structure: findings, repair plans, historical trends
Why it's "deliberately honest": never inferring usage from configuration presence
Installation and usage in Claude Code and Codex

Prerequisites

Experience using Claude Code, Codex, Cursor, or a similar AI coding tool
Familiarity with AGENTS.md, Hooks, and Skills helps but isn't required
Basic awareness of software quality practices (CI/CD, testing, code review)

Project Background

The Problem: Agents Move Fast, Workflows Stay Weak

AI coding agents introduce a new failure mode: speed. An agent can finish in minutes what previously took hours — but that speed also skips the slow steps that were actually valuable. Careful goal understanding, working within proven paths, validating changes, passing human review.

Better Harness identifies five common workflow failure patterns:

Failure Pattern	How It Shows Up
Fuzzy goals	Agent doesn't know what "done" looks like; keeps iterating in the wrong direction
Improvised execution	Starting from scratch each time; no reusable execution paths
Unvalidated changes	Code got modified, but no evidence confirms the modification worked
Bypassed safeguards	AI speed made quality gates optional
Lost lessons	Insights from this session don't help the next one

These five problems are typically invisible in code diffs. The code passes review. The workflow problems persist and repeat on the next task.

QoderAI and Qoder

Better Harness is built by QoderAI, who also makes a desktop AI coding agent called Qoder. Better Harness is natively integrated in Qoder; the open-source version runs as a plugin in other major agents.

Project Stats

⭐ GitHub Stars: 1,500+
🍴 Forks: 123+
📄 License: MIT
Runtime: Node.js 22.20.0–25.0.0

Core Concept: Feedforward + Feedback Dual Signals

Better Harness's evaluation model rests on one framework: effective agent workflows need two signal types working together.

Before work starts                After / during work
──────────────────                ──────────────────
Feedforward guides                Feedback sensors

AGENTS.md                         Linters
Spec documents                    Test suites
Skills (reusable steps)           Hooks (event-triggered)
Acceptance criteria               Evaluation agents
                                  Diagnostics

Feedforward guides steer the agent before it acts — AGENTS.md establishes rules and goals, specs define task scope, Skills provide proven execution paths, acceptance criteria define what "done" looks like.

Feedback sensors observe results after the agent acts — linters check conventions, test suites validate functionality, Hooks capture signals on trigger events, evaluation agents score output quality.

The core indicator of a healthy workflow: both sides are operating, and their results leave evidence.

The Five Dimensions

Dimension 1: Task Understanding

Core question: Does the agent know what the goal is and what "done" looks like?

What gets evaluated:

Whether AGENTS.md exists and contains effective rules and goal definitions
Whether spec documents define task scope
Whether explicit acceptance criteria tell the agent when to stop
Whether the agent can identify the project starting point and appropriate change granularity

Common failure: Vague goal descriptions ("improve login flow" vs. "add specific error messages for each error state") leave the agent without a clear stopping point, producing over-engineering or repeated course corrections.

Dimension 2: Controlled Execution

Core question: Is the agent working on supported, repeatable paths?

What gets evaluated:

Skills configuration: whether reusable SDLC execution steps exist
MCP tool availability and boundary settings
Sandbox boundaries: whether agent permissions are appropriately scoped
Whether the agent uses proven paths rather than improvising each time

Common failure: Reinventing the execution process for each task; agent with excessive permissions making changes beyond task scope; no reusable steps, so similar tasks produce inconsistent quality.

Dimension 3: Change Validation

Core question: Is there evidence the change actually works?

What gets evaluated:

Whether tests actually ran after the change (not just "tests exist")
Whether lint checks actually ran after the change
Whether Hooks captured validation signals
Whether re-validation happened after validation failures
Whether diagnostic tools were actually used

Key distinction: Better Harness separates "tests are configured" from "tests were executed." A project can have a complete test suite, but without evidence the agent ran tests after changes, this dimension doesn't score for that.

Dimension 4: Reliable Delivery

Core question: Did AI speed bypass quality gates?

What gets evaluated:

Whether task acceptance has verifiable evidence (not just "code is done")
Whether high-risk operations have human approval paths
Whether rollback mechanisms exist
Whether CI/CD pipelines are part of the agent's workflow
Whether human review actually happened

Core concern: An agent can make extensive modifications without anyone noticing. Reliable Delivery evaluates what validation gates those modifications passed before delivery.

Dimension 5: Learning Capture

Core question: Do lessons from this task improve the next one?

What gets evaluated:

Whether recurring issues get distilled into reusable Rules or Skills
Whether Loop Discovery is working (pattern recognition generating suggestions)
Whether the Memory system is in use
Whether similar tasks reuse existing experience or start from scratch each time

One signal: Better Harness flags "long sessions" (over 45 minutes) for human review — these often indicate the agent spent significant effort on exploration that should be captured to avoid repetition.

Analysis Architecture: Three Independent Evidence Agents

Better Harness doesn't use a single agent for all analysis. Three independent read-only sub-agents collect different evidence categories in parallel, and a lead agent synthesizes the results only after independent collection completes.

Three independent sub-agents (parallel)
├── Agent 1: Customization asset analysis
│       → Completeness of Rules, Skills, Hooks, and configs
│
├── Agent 2: Real task session analysis
│       → What the agent actually did and how it performed
│
└── Agent 3: Project engineering foundation analysis
        → Whether project structure supports the agent workflow

        ↓ (after independent collection)

Lead Agent: Unified analysis + report generation

Why keep them independent: Running the three sub-agents separately prevents one category's conclusions from skewing another's interpretation. If Agent 1 finds complete Skills configuration, that shouldn't influence how Agent 2 reads the actual session records — Agent 2 looks at execution evidence only.

Missing evidence handling: Unobserved behavior is never inferred. No test execution records → Change Validation is unknown, not assumed based on the presence of test files in the project.

Report Structure

Running the analysis produces three files:

report.html: Self-contained visual report (open directly in a browser)
report.md: Markdown format for version control and team sharing
findings.json: Structured data for programmatic processing

What the Report Contains

Five-dimension overview: Scored bar chart per dimension + count of related findings

Scope snapshot: Current configured asset inventory — Rules count, Skills count, custom Agents, MCP tools, Memories, Hooks

Prioritized findings: Each finding includes:

Priority (High / Medium / Low)
Owning dimension
Cause: the specific gap in current configuration
Expected Output: what a successful fix achieves
Fix instructions: an editable pre-filled prompt starting with /harness

Session observations: Representative patterns extracted from analyzed sessions; sessions over 45 minutes flagged separately

Historical trend: Results compared across multiple runs, showing dimension-level change over time

Deliberately Conservative Scoring

Better Harness has an explicit scoring constraint:

"Configured assets can establish that a mechanism exists, but only linked task evidence can establish that it was used."

A project with complete Skills configuration doesn't get full marks on Controlled Execution without evidence of actual usage. Passing a current check proves the intervention was exercised; only a comparable later result can prove the loop improved. The history view shows recorded trends, not causal proof of improvement.

Installation and Usage

Claude Code

/plugin marketplace add QoderAI/better-harness

Other Platforms

Platform	Install Method
Codex Desktop	Settings > Plugins > Add from Marketplace
Codex CLI	`codex plugin marketplace add [repo URL]`
GitHub Copilot	`copilot plugin marketplace add QoderAI/better-harness`
Qwen Code	`qwen extensions install QoderAI/better-harness`
Cursor	Clone repo locally, source-local install
Qoder	Natively built in — no install needed

Run Analysis

After installation, in any supported agent:

/better-harness analyze this project's AI coding workflow and generate an evidence-backed report

Outputs a self-contained report.html + report.md + findings.json.

The Repair Workflow

Better Harness never modifies anything directly — it identifies gaps and provides the fix starting point:

High-priority finding → click "Plan a fix"
    ↓
Fix detail opens:
  - Cause: the specific config gap
  - Expected Output: what a fix achieves
  - Fix instructions: editable pre-filled prompt
    ↓
Click "Start Fix" → launches as a Quest task
    ↓
Agent executes fix in an inspectable, reversible Quest task
    ↓
Re-run /better-harness → confirm the loop actually improved

Fix results can be distilled further into Rules, Skills, and Memories, letting subsequent tasks benefit directly.

Resources

🌟 GitHub: QoderAI/better-harness
📖 Docs: docs.qoder.com/user-guide/knowledge-engine/better-harness

Summary

Better Harness solves a meta-level problem: an AI coding agent's output quality depends on the surrounding workflow, not just the model's capability. The same Claude Sonnet in a workflow with complete AGENTS.md, clear acceptance criteria, post-change test runs, and experience distilled into Skills produces significantly better outcomes than in a workflow without any of these.

The five-dimension framework makes "workflow health" measurable — not "the workflow feels off" but "Change Validation scored low because no test execution records were found." Priority ordering tells you what to fix first. Repair plans tell you how. The historical trend confirms fixes actually worked.

The deliberately conservative scoring is what makes this tool trustworthy. It doesn't infer "tests ran" from "test files exist." It doesn't claim "the repair caused the improvement" from historical score increases. That honesty means the output can be acted on rather than second-guessed.

If you're using Claude Code, Codex, or Cursor and your sessions occasionally feel inefficient — goals that took too long to clarify, changes that broke unexpectedly, the same problems recurring — /better-harness surfaces which part of the loop is actually broken.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Codebase Knowledge Base Series (08): Production Architecture — How to Combine Vector, Graph, and Symbol Indexes

WonderLab — Thu, 06 Aug 2026 01:58:35 +0000

We Spent Five Articles Measuring the Boundary — Now We Need a Bigger Map

If you've read straight through from Article 03, then together we've laid to rest a ghost that haunted us for five full articles — Q8.

Let's recap the long chase. Q8 is process payment and create Stripe charge, and its ground truth includes calculate_order_total. We threw every trick in the text-retrieval book at catching it: the vector baseline couldn't; three chunking strategies couldn't; encoding called_by into the embedding only nudged similarity to 0.51 and still couldn't; and finally even the industry's universally endorsed "ultimate weapon" — BM25 + vector hybrid search — crashed, not only failing to fix Q8 but dragging the total from 0.958 down to 0.931.

The five-article conclusion condensed into a single sentence:

The link between calculate_order_total and "Stripe payment" is purely structural. It lives on an edge in the call graph (it's called by process_checkout), and is written in no function's text. Therefore any pure-text approach — vector, BM25, hybrid — physically cannot reach it.

That sentence is a hard, number-backed conclusion we bought with five articles of experiments. It's valuable — but it's only one piece of the puzzle.

Because while chasing Q8, we incidentally mapped out the territory of three distinct retrieval routes: what vector is good at and what it can't reach; what graph is good at and what it costs; and a whole class of query — "where is the function named validate_jwt_token" — that needs no semantics at all, where exact matching returns in a millisecond. Stitch those three territories together and you get the complete map of codebase retrieval.

So this article shifts altitude. The previous five held a magnifying glass to the recall of a single retrieval algorithm; this one pulls back to satellite view to draw a system architecture: how the vector, graph, and symbol signals should actually be organized into a working production system.

A heads-up to keep in mind before you read: anything in this article with a concrete Recall number is a conclusion already validated by experiments in the previous five articles; anything describing "how to design the system" is an engineering design recommendation derived from those conclusions, not yet validated end-to-end on a full production system. I'll flag this repeatedly at the key points — don't mistake design recommendations for experimental conclusions.

First, Draw the Three Signals' Territories Clearly

Before designing anything, let's put the three territories the previous five articles measured out on the table. This is the bedrock of the entire architecture, and every brick in that bedrock carries an experiment number.

Route one: vector retrieval — the home turf of semantic-similarity queries.

This is the baseline established in Article 03 and repeatedly validated in the four that followed: AST function-level chunking + raw-code embedding, Recall@5 = 0.958, the strongest vector baseline among all text approaches (experimentally validated). It excels at queries of the form "describe a capability in natural language, find the implementation" — say "find the function that encrypts and securely stores passwords" and it hits hash_password, even if you never mentioned the function name.

Its boundary is measured just as clearly: the semantic gap can't be filled (Articles 06, 07). calculate_order_total and "Stripe payment" are genuinely different things in the real world, and no embedding trick can pull them closer.

Route two: graph retrieval — the home turf of structural-relationship queries.

Article 05 proved that the structural information carried by the call graph is an orthogonal truth vector can't reach (experimentally validated): Q8 was scooped back in two hops via the process_checkout → calculate_order_total call edge. But Article 05 also measured its cost — naive BFS 2-hop expansion bloats the candidate set, squeezing Q1's correctly-hit verify_password out of the top-5, one fixed and one broken, netting zero.

The conclusion is subtle: right direction, crude method. The graph signal is valuable, but "doing naive BFS expansion after retrieval" is the wrong way to use it. This lesson directly determines graph retrieval's role in the architecture below.

Route three: symbol retrieval — the home turf of exact-match queries.

The previous five articles didn't run dedicated experiments on this route, but its value is self-evident and needs no proof: when your query is itself a precise symbol — "where is the function named validate_jwt_token defined," "which files import redis.Redis" — you don't want "semantically close," you want "literal exact hit." Here vector is a sledgehammer for a nut, and easily led astray by semantic approximation; a grep or a function-name → file → line-number symbol table returns precisely in under 10 milliseconds.

Place the three territories side by side and a beautiful fact emerges — they barely overlap:

Query type                          Best route    Example
────────────────────────────────  ──────────  ─────────────────────────
NL capability description, find impl   Vector     "find the function that validates JWT"
Exact symbol name / import             Symbol     "where is validate_jwt_token"
Structural relation / call chain       Graph      "who calls createPayment"
────────────────────────────────  ──────────  ─────────────────────────

The "four knowledge layers" framework from Article 01 maps neatly onto this: syntax layer (AST/symbol), semantic layer (vector), architecture layer (graph), plus the intent layer (Git history). The first principle of a production system is to accept that these four signals are each independent and none is dispensable — there is no single "best retrieval method" that dominates all four layers.

Principle One: Multi-Path Retrieval, Each to Its Own Job

The first principle, and the most counterintuitive: stop looking for "the one best retrieval method."

That's exactly what we did for five articles — endlessly asking "does vector work? does BM25 work? does hybrid work?", each time betting one route could dominate every query. Five straight losses later, we were forced to admit: for structural queries like Q8, the text route physically can't reach.

Flip the framing: not one all-rounder, but let each route do only what it's best at. Vector for semantics, graph for structure, symbol for exact matching — three routes retrieve in parallel, each covering its own territory.

An analogy: this is like a hospital's triage desk. You wouldn't expect one general practitioner to handle heart bypass, tooth extraction, and blood tests all at once. You triage first — fractures to orthopedics, toothaches to dental, bloodwork to the lab. Each department is an expert in its own domain and an amateur outside it. Retrieval is the same — using vector for exact symbol matching is like having your dentist do the heart bypass; sure, he technically knows what a heart is, but you really don't want him operating.

This principle flows directly from the previous five articles' lessons: we've already used five articles' worth of failures to prove what happens when you force one signal to do a job it's bad at. Vector was forced to reach Q8's structural relationship, tried for five articles, and never reached it. So let graph do that job and let vector return to its semantic home turf.

Flag: "Multi-path retrieval" as an architectural direction is a design principle derived directly from the experimental conclusions of Articles 05/07. Each route's territory is experimentally backed (vector 0.958, graph fixing Q8, symbol needing no experiment), but "the overall Recall after combining three routes into a full system" has no end-to-end experimental data yet — this is a design recommendation, not an experimental conclusion.

Principle Two: Query Routing, Activate the Right Path by Intent

With three retrieval routes, a question immediately arises: when a query comes in, which routes do we activate?

If we blindly run all three every time and then fuse, it's not just slow — the routes pollute each other. Article 07 demonstrated this graphically: vector scored a perfect 1.00 on Q7, but BM25 dragged it to 0.67 by over-matching the high-frequency generic word "execute," and once RRF fused them, the good route got pulled down with the bad. Blind fusion lets the route that's bad at a query drag down the route that's good at it.

So we need a Query Router: first judge what type the query is, then decide which routes to activate.

Query "where is the validate_jwt_token function"   → Symbol route (exact match)
Query "the function that validates JWT tokens"      → Vector route (semantic)
Query "what does createPayment call"                → Graph route (structure)
Query "the full payment flow call chain"            → Graph route + Vector route (mixed)

Implementation doesn't require reaching for an LLM up front. The vast majority of query intents can be recognized by a handful of rules:

The query contains a precise identifier (a snake_case/camelCase function name, an imported module name) → prefer the symbol route.
Structural keywords appear — "who calls," "what does it depend on," "call chain," "full flow" → activate the graph route.
Everything else, natural-language description → the vector route.
Compound queries (both structural intent and semantic description) → multiple routes jointly.

Edge cases the rules can't classify can fall back to a small model for assistance — but that's not a necessity; rules cover the overwhelming majority of scenarios.

Flag (important): The "query routing" here is a pseudocode-level design concept, not an experimentally validated result. The "query → route" mapping table above is a set of routing rules I hand-designed based on the three territories; it aligns with the previous five articles' experimental intuition, but questions like "how accurate is the routing, how costly is a misroute" have not been experimented on in this series. Please read it as a design recommendation.

Principle Three: Graph Retrieval Is a First-Class Citizen, Not a Post-Processing Bolt-On

This is the single most important shift in the whole architecture, and a direct correction of Article 05's failure.

Recall why Article 05 crashed: its flow was vector-retrieve the top-3 seeds first, then BFS-expand the graph from the seeds. Graph traversal was a post-processing step trailing behind vector. That order was wrong — it treated graph as a patch on vector, letting the "structurally related but query-irrelevant" functions dragged in by graph expansion pollute vector's ranking and break Q1.

The right approach promotes graph retrieval to a first-class retrieval route on equal footing with vector:

The graph index and vector index are built in parallel, both independent first-class retrieval entry points, neither depending on the other.
The vector route takes its own top-k; the graph route executes independently — it identifies the function/module names mentioned in the query and directly traverses the graph (along CALLS/CALLED_BY edges), returning structurally related functions.
The two routes' results are fused at the end. Note this fusion is not the context-blind RRF of Article 07, but a weighted merge by query type: the graph route weighs high for structural queries, the vector route weighs high for semantic ones.

One diagram makes the shift clear:

    ❌ Article 05 (graph as a post-processing patch)
    Query → Vector top-k → BFS expand → rerank
                            ↑ graph here, trailing vector, polluting the ranking

    ✅ Production architecture (graph as an equal first-class citizen)
    Query ─┬─→ Vector route ─┐
           └─→ Graph route  ─┴─→ weighted fusion by query type → results
              two routes in parallel, each retrieving independently

Why does this change solve Article 05's problem? Because Article 05's root cause was "graph expansion bloated vector's candidate set and diluted vector's ranking." When the graph route becomes an independent route with its own retrieval logic and trigger conditions (Article 07's ending recommended: only trigger on high-confidence functions, only 1 hop, filter neighbors by business rules), it no longer indiscriminately stuffs noise into vector's candidate pool. The graph route scoops back Q8's calculate_order_total, the vector route holds Q1's verify_password, each minds its own business, no interference.

Flag: "Graph as a first-class citizen, weighted fusion by query type" is a design-correction direction derived from the failures of Articles 05/07. Article 05's "graph post-processing pollutes ranking" is an experimentally validated failure; but "parallel first-class citizen + weighted fusion can hold both Q1 and Q8 at once" is a design inference, not yet validated by experiments in this series. The direction has experimental backing; the specific fusion weights and trigger conditions need the upcoming hands-on article to tune.

Principle Four: Incremental Updates, Never Full Rebuilds

The first three principles solve "how to query"; the fourth tackles a more brutal engineering reality: code changes every day.

The previous five articles all ran on a static 28-function toy dataset. But a real project isn't a specimen — dozens to hundreds of commits a day, functions added/removed/modified, signatures changing, call relationships rewiring. If the indexing strategy is "full rebuild every time," then for a codebase of hundreds of thousands of lines, building the vector index alone takes tens of minutes, and the graph and symbol table have to be fully recomputed along with it. A developer waiting ten minutes for re-indexing after changing one line — no one will use that system.

So the fourth principle: Git diff-driven incremental updates, never full rebuilds.

Three specifics:

1. Git diff-driven. When a commit comes in, first compute exactly which files and functions it changed, and re-index only those change points. For untouched functions, not a single byte of their embedding, graph nodes, or symbol-table entries needs recomputing.

2. Change propagation. This one is easily overlooked but crucial. A function change isn't isolated — if create_payment_intent's signature changes, then all functions that call it need their call-graph edges updated too. So incremental updates aren't just "recompute changed functions" — they must also propagate the change to affected neighbors along the call graph. This is yet another benefit of treating graph as a first-class citizen: the graph structure itself is the propagation path.

3. Version snapshots. Support querying historical versions by commit hash — which connects to Article 01's fourth layer, the intent layer. "What did this function look like three months ago," "which commit introduced this line and what problem was it solving" — these queries want not the current code but the code's evolution history, and the answer lives in Git.

    Code Change (git commit)
             │
       ┌─────▼──────┐
       │  Git Diff  │   ← find only changed files / functions
       └─────┬──────┘
             │
       ┌─────▼──────┐
       │ AST Parser │   ← re-parse only changed files
       └──┬──────┬──┘
          │      │
      ┌───▼──┐ ┌─▼──────┐
      │Embed │ │ Graph  │   ← two routes update incrementally in parallel
      │Update│ │ Update │      Graph also propagates changes
      └──────┘ └────────┘      to neighbors along call edges

Flag: Incremental updating is a pure design recommendation — all five prior articles experimented on a static dataset and ran no incremental-update experiments whatsoever. But this direction is uncontroversial — it's standard equipment for any production-grade indexing system, and a direct response to the "dynamism is the biggest challenge" point Article 01 already made.

The Complete System Architecture

Assemble the four principles and you get a complete system architecture. View it from two angles: what happens at query time, and what happens at index time.

Query flow (online, when a user issues a query):

                        Query
                          │
                   ┌──────▼──────┐
                   │Query Router │  ← recognize query intent (rules first)
                   └──┬───┬───┬──┘
                      │   │   │
              ┌───────▼┐ ┌▼──────┐ ┌▼────────┐
              │ Vector │ │ Graph │ │ Symbol  │
              │ Index  │ │ Index │ │ Index   │
              │(AST +  │ │(CALLS/│ │(grep /  │
              │embedding│ │CALLED_│ │AST      │
              │)       │ │BY)    │ │symbols) │
              └───┬────┘ └──┬────┘ └────┬────┘
                  │         │           │
              ┌───▼─────────▼───────────▼────┐
              │        Result Merger          │
              │  (weighted merge by query     │
              │   type, deduplicate)          │
              └──────────────┬────────────────┘
                             │
                        Top-k Results

Three index routes retrieve in parallel (the Router decides which to activate), then the Result Merger merges by query-type weighting, deduplicates, and produces the final results. Note the Merger isn't a blind RRF — it knows whether this is a structural or a semantic query and adjusts each route's weight accordingly. This is exactly what Article 07 taught us: fusion must be context-aware, or the good route gets dragged down by the bad.

Index flow (offline, Git hook triggered):

That's the Git diff-driven incremental-update diagram from the previous section. The three index routes update incrementally in parallel on commit, with the graph index additionally handling change propagation.

Put the two diagrams together and you have the full picture of this architecture: at query time, three routes retrieve in parallel and fuse with intent-aware weighting; at index time, Git diff drives three routes' parallel incremental updates. Every design decision traces to an experimental lesson from the previous five articles.

Implementation Complexity of Each Index Layer

Design is design; when it comes to landing it, you need to know each route's engineering cost. The table below lays out the three routes (plus the intent layer): build cost, update cost, query latency, applicable scenarios.

Index type	Build cost	Update cost	Query latency	Applicable queries
Symbol index (grep / AST symbols)	Low (seconds)	Very low (incremental)	< 10ms	Exact symbol queries
Vector index (AST + embedding)	Medium (minutes)	Medium (changed funcs only)	~100ms	Semantic queries
Call-graph index (AST parse)	Low (seconds)	Low (changed files only)	< 50ms	Structural queries
Git history index	High (initial full scan)	Low (incremental commit)	Varies	Intent / history queries

A few things worth noting:

The symbol index is the value king. Seconds to build, 10ms to query, near-negligible cost, yet it cleanly covers an entire class of exact queries. Any system should ship it first.
The vector index is the most expensive route. Embedding runs a model, build is measured in minutes, and query latency is highest (~100ms). This is exactly why incremental updates matter most for it — you absolutely don't want to re-run full embedding on every commit.
The call-graph index is surprisingly cheap. It's just AST parsing plus an edge table, seconds to build, fast to query. Article 05 already proved its value, and its cost is so low there's no reason not to ship it. Graph retrieval has been neglected not because it's expensive, but because people never figured out how to use it (Article 05's lesson).
The Git history index is priciest on the first run. A full scan of commit history isn't cheap, but every new commit afterward is incremental, with low marginal cost.

Flag: The costs and latencies in the table are order-of-magnitude estimates (seconds/minutes/milliseconds), drawn from the actual run experience of the previous five demos and ordinary engineering experience, not precise benchmarks on production-scale codebases. Use it to judge relative priority (ship the cheap symbol and graph routes first, the expensive vector route later), don't treat it as an SLA.

Implementation Path: Small to Large, Three Phases

No matter how pretty the architecture, shipping it all at once will choke you. The right posture is a phased rollout where each phase delivers standalone value; get one working before moving to the next.

Phase 1: Vector + Symbol (minimum viable system)

Ship the two cheapest, least controversial routes first:

Symbol index: ripgrep + an AST symbol table (function name → file → line-number dictionary). Seconds to build, covers all exact queries.
Vector index: AST function-level chunking + raw-code embedding. This is the strongest vector baseline validated in Article 03 (Recall@5 = 0.958, experimentally validated), copy it directly.
Query routing: start with the simplest rule — precise identifier in the query goes to symbol, otherwise vector.

This phase already covers the two big classes of "exact symbol query" and "semantic description query," a system usable immediately. Don't underestimate it — the vast majority of daily code retrieval falls into these two classes.

Phase 2: Add the call graph (cover structural queries)

Once Phase 1 runs smoothly, add the third route:

Call-graph index: AST parsing to build CALLS/CALLED_BY edge tables. Low cost (seconds), high value (Article 05 validated it fixes Q8).
Wire the graph route in as an independent first-class retrieval route — identify function names from the query, traverse the graph independently, absolutely no Article-05-style "BFS trailing behind vector" post-processing.
Upgrade the query router: recognize structural keywords like "who calls," "call chain," and activate the graph route.
Upgrade the Result Merger: from "single-route passthrough" to "weighted fusion by query type."

This phase fills in the Q8 class of structural queries — the piece of the map the text route physically can't reach.

Phase 3: Add Git history + incremental updates (productionize)

The first two phases are "can query accurately"; this phase is "can survive production":

Git history index: connect to Article 01's intent layer, support queries like "why is this line written this way," "the version three months ago."
Incremental updates: refactor all three routes' index building to be Git diff-driven. This is the critical leap from "demo" to "production" — without incremental updates, the architecture above won't run on a real project.
Change propagation: when a function signature changes, update affected neighbors' edges along the call graph.

The point of phasing is that each phase is a complete system that can ship independently and create value independently. Phase 1 solves most retrieval needs; Phase 2 covers the structural blind spot; Phase 3 lets it survive a real project's daily evolution. You needn't wait for all three phases to deliver — on the contrary, you should collect real queries after Phase 1 ships and use them to guide the priorities of Phases 2 and 3.

Tool Selection Recommendations

Finally, down to the actual screws. This part is conventional engineering selection advice, not experimental conclusions.

Vector storage: pgvector (a PostgreSQL extension) or Qdrant. The key reason to pick them is support for metadata filtering — you can attach conditions like "only in the payment module," "only in these few files" during vector retrieval, which is extremely useful for code retrieval (many queries naturally carry a module scope).

Graph storage: no Neo4j needed. This deserves emphasis, because the moment you say "knowledge graph," many people reflexively reach for a graph database. Every demo's call graph in this series is a Python dict ({function_name: [called functions]}) held in memory and serialized to a file, and that's entirely sufficient (validated in Article 05). For a large project where memory won't hold it, NetworkX or simply an edge table in SQLite is still far lighter than Neo4j. The scale of a code call graph can't justify the operational cost of a dedicated graph database.

Symbol index: ripgrep (version 15.1.0 supports PCRE2 + JIT, fast enough) for full-text exact matching, plus an AST-parsed symbol table (function name → file → line number) for definition-level queries. The two together cover all exact-symbol needs.

Incremental-update triggers: a Git pre-push hook, or an index-update step in the CI/CD pipeline. The former updates instantly and locally; the latter is centralized, suited to a team-shared index.

Query router implementation: rules first (keyword recognition + identifier pattern matching), LLM assistance second (and non-essential). Don't reach for LLM classification up front — rules cover the overwhelming majority, and they're fast, free, and explainable.

A Ready-Made Landing Case: codebase-memory-mcp

After all this design talk, you might ask: is there anything running that implements this architecture? Yes.

The codebase-memory-mcp MCP Server introduced in Article 02, held up against this article's architecture design, is essentially a complete landing case:

It already implements vector retrieval + call graph + symbol index, three-route retrieval — exactly this article's Principle One, multi-path retrieval.
It's exposed directly to Claude Code via the MCP protocol — within Claude Code you can directly call its search_graph (symbol/semantic search), trace_path (call-chain tracing), query_graph (graph queries), and other tools.
It organizes the three signals into a unified knowledge-graph interface — exactly this article's idea of "combining three routes into one system."

In other words, the architecture diagram in this article isn't armchair theory — it has an implementation that already works and is already wired into the Claude Code workflow. In the next hands-on article, we'll take codebase-memory-mcp and run an end-to-end demo on a real project, to see how this multi-path retrieval architecture actually performs on a real codebase — and that's when all the experimental intuition accumulated over the previous five articles finally faces the test of a real project.

Summary

Five experiments measured the three signals' territories, and they barely overlap. Vector for semantic-similarity queries (0.958, validated), graph for structural queries (fixing Q8, validated), symbol for exact-match queries (no experiment needed). The first principle is accepting there's no "best method" that dominates all four layers.
Principle one: multi-path retrieval, each to its own job. Stop looking for an all-rounder, let each route do only what it's best at — a design direction derived from five articles of consecutive failure at "forcing one signal to dominate."
Principle two: query routing, activate the right path by intent. Precise identifiers to symbol, structural keywords to graph, natural language to vector, rules first and LLM second. (Pseudocode-level design concept, not experimentally validated.)
Principle three: graph retrieval is a first-class citizen, not a post-processing bolt-on. Directly correcting Article 05's failure — graph and vector retrieve independently in parallel, fused by query-type weighting, rather than letting graph trail behind vector doing BFS and polluting the ranking. (Direction experimentally backed, fusion details to be validated hands-on.)
Principle four: Git diff-driven incremental updates, never full rebuilds. Recompute only changed functions, propagate changes along the call graph, support commit version snapshots connecting to the intent layer. (Pure design recommendation, responding to Article 01's "dynamism is the biggest challenge.")
Land it in three phases, each delivering standalone value. Phase 1 vector+symbol (minimum viable) → Phase 2 add call graph (cover structural blind spot) → Phase 3 add Git history + incremental updates (productionize).
Tool selection: pgvector/Qdrant for vectors (need metadata filtering), no Neo4j for graph (a Python dict / SQLite edge table suffices), ripgrep + AST symbol table for exact matching, Git hook or CI to trigger incremental updates.
codebase-memory-mcp is a ready-made landing case for this architecture, already implementing three-route retrieval and wired into Claude Code. The next article uses it for an end-to-end hands-on demo on a real project.

From Article 03's single vector baseline to this article's three-route architecture, we've completed the shift in vantage point from "retrieval algorithm" to "system architecture." The sentence Q8's story taught us is the very bedrock of this architecture:

Some links are inherently not in the text — so what we've always needed isn't a smarter single route, but multiple routes each doing its own job.

References

codebase-memory-mcp tool introduction: see Article 02 of this series
The four knowledge layers framework: see Article 01 of this series
The graph-retrieval double-edged-sword experiment: see Article 05 of this series
The text-route boundary experiment: see Article 07 of this series

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #142: Buzz — Block Rebuilds Team Collaboration on Nostr, Where AI Agents Hold Their Own Cryptographic Identity

WonderLab — Thu, 06 Aug 2026 01:57:22 +0000

Introduction

"Not agents assisting humans. Agents and humans in the same room, working on the same thing."

This is article #175 in the "One Open Source Project a Day" series. Today's project is Buzz — the open-source team workspace released by Block (Jack Dorsey's company) on July 21, 2026. Its core premise: AI agents should join teams as full members, not be bolted on as bots or plugins.

Slack, Teams, GitHub — existing collaboration tools integrate AI roughly like this: configure a bot token, let the bot monitor a channel, trigger a response when @mentioned. The agent has no independent identity, no persistent history, no auditable action record, and permissions controlled by the platform rather than the organization.

Buzz takes a different approach. Built on the Nostr protocol, every participant — human or agent — holds a keypair that belongs to them, not the platform. Identity, behavior, and history are all cryptographically signed and portable to any Nostr-compatible system. A human member and an agent member look identical in the channel.

21,300 Stars. Apache 2.0. That's six days after launch.

What You'll Learn

Why Buzz chose Nostr instead of building its own identity system
How agents become full members in Buzz, not bots
How the ACP protocol drives Claude Code, Codex, Goose, and other agents
buzz-cli's machine-first design: built for LLM tool calls
The YAML workflow engine: triggers, steps, and action types
The Rust + Axum backend, Tauri desktop app, full technical stack

Prerequisites

Familiarity with team collaboration tools (Slack/Discord-type workflows)
Basic understanding of AI agent tool-calling
Knowing what a keypair (public/private key) is helps

Project Background

Why Block Built Buzz

Block spent two years building AI tools internally and reached one conclusion: productive work happens when humans and agents are in the same context together. Separate tools, separate systems, separate histories produce fragmentation and repeated context-rebuilding.

The problem with existing platforms isn't missing features — it's the wrong design assumption: they treat AI as "assistants" rather than "members."

Buzz's starting point: agents should have the same participation rights as human members — independent identity, auditable behavior, persistent history, configurable permissions.

Timeline

July 21, 2026: Public launch with simultaneous open-source release
GitHub: 21,300 Stars within days of launch
Hosted: buzz.xyz (Block's instance)
Self-hosted: Full Docker + Rust toolchain support

Author / Team

Company: Block, Inc. (founded by Jack Dorsey)
License: Apache-2.0
Primary language: Rust (backend) + TypeScript/React (desktop via Tauri) + Dart (mobile via Flutter)

Project Stats

⭐ GitHub Stars: 21,300+
🍴 Forks: 2,300+
📄 License: Apache-2.0
📅 Launch date: 2026-07-21

Why Nostr

Buzz's identity system runs entirely on the Nostr protocol. That decision shapes the entire product architecture.

The Problem Nostr Solves

Traditional collaboration platform identity: the platform issues accounts, controls access, owns the identity and history. Switching platforms means rebuilding everything.

Nostr's model: each participant generates their own secp256k1 keypair. The private key belongs to them. Every message is signed with that key. Identity belongs to no platform, history is portable, behavior is non-repudiable.

What this means for agents:

Traditional bot model:
  Platform issues bot token → agent calls API with token → token revoked = agent gone
  Identity belongs to the platform, history belongs to the platform

Buzz / Nostr model:
  Agent generates its own keypair → signs every message with its private key → key belongs to agent
  Identity is portable, history lives in the relay, switching relays preserves identity

Nostr NIPs in Use

NIP-01: Base event format (the underlying structure of every message)
NIP-42: Authentication (verifying key ownership when connecting to the relay)
NIP-34: Git integration (patch submissions, repo announcements, commit status)

"The Relay URL Is the Community"

One Buzz design decision: a community (workspace) is identified by its relay URL. wss://your-org.buzz.xyz is your workspace — the URL itself is the authoritative identifier. Different relays are different communities. All state on one relay shares a single event log.

How Agents Become Full Members

Not a Bot, a Member

Traditional platform bots: a service account that monitors specific channels and triggers responses on @mention. In the permission model, typically a separate category from human members.

Buzz agents: hold their own keypair, join channels exactly as humans do, and their messages are signed Nostr events that appear in channel history indistinguishably from human messages.

Adding an agent to a channel:

# Same operation used for adding a human member
buzz channels add-member --channel CHANNEL_ID --pubkey AGENT_PUBKEY --role member

What Cryptographic Identity Means in Practice

Every message an agent sends carries a signature from that agent's private key:

Non-repudiable: anyone can verify which agent produced any given message
Auditable: agent actions in the audit log have unambiguous attribution — not "some bot"
Portable: the agent's keypair isn't bound to Buzz; the same identity works on any Nostr-compatible system

How ACP Drives Agents

Buzz routes relay events to agent processes through ACP (Agent Client Protocol):

Buzz relay (WebSocket event stream)
        ↓
buzz-acp (standalone binary, runs outside the relay)
        ↓
ACP (JSON-RPC over stdin/stdout)
        ↓
Agent process (Claude Code / Codex / Goose / custom)
        ↓
MCP tool calls → message posted back to channel

Key design: one active prompt per channel at a time. Subsequent @mentions queue while the current prompt is in flight, preventing overlapping context windows from race conditions. Agent process crashes trigger automatic harness restarts. Conversation history lives in the relay's Postgres database, not the ACP process — agents restart with full context intact.

Supported Agents

Built-in support (auto-detected from locally installed harnesses):

Claude Code (Anthropic)
Codex (OpenAI)
Goose (Block's own agent)
Grok (xAI)
Any custom ACP-compatible agent

buzz-cli: An Interface Built for Machines

Machine-First Design Principles

The Buzz CLI is designed for programmatic consumption, not human convenience:

JSON to stdout: all output is structured JSON, parseable by programs
Structured errors to stderr: error output is separated from result output
Meaningful exit codes: 0 = success, non-zero = typed failure, scriptable

Configuration via environment variables:

export BUZZ_RELAY_URL='http://localhost:3000'
export BUZZ_PRIVATE_KEY='nsec...'   # agent's or user's private key

Core Operations

# Channel management
buzz channels list
buzz channels create --name "dev-ops"

# Send a message
buzz messages send --channel CHANNEL_ID --text "Deploy complete, v2.3.1"

# Full-text search (covers messages + canvases + git events — all one index)
buzz messages search --query "deploy failed" --since 24h

# Read a thread
buzz threads read --thread-id THREAD_ID

# Canvas read/write (documents)
buzz canvases write --canvas-id ID --content "..."

# Workflow management
buzz workflows list
buzz workflows run --workflow-id ID

# Agent memory storage
buzz memory set --key "deployment-config" --value "..."
buzz memory get --key "deployment-config"

# Git repo announcements (NIP-34)
buzz repos announce --url https://github.com/org/repo

The intended use: agents call these commands via MCP tools to post reports to channels, read context, update canvases, and query history — without directly manipulating the relay protocol.

YAML Workflow Engine

Trigger Types

Workflows support four trigger types:

trigger:
  on: message_posted       # when a message is posted
  # on: reaction_added     # when someone adds an emoji reaction
  # on: schedule           # cron expression
  # on: webhook            # external HTTP webhook

Full Example: Release Request Workflow

name: "Release request"
trigger:
  on: message_posted
  filter: "str_contains(trigger_text, 'ship it')"
steps:
  - id: announce
    action: send_message
    channel: "{{trigger.channel_id}}"
    text: "Release requested by {{trigger.author}} — starting deployment pipeline."

  - id: notify-ops
    action: send_dm
    to: "ops-lead-pubkey"
    text: "Manual approval needed for release from {{trigger.author}}"

  - id: wait-approval
    action: wait_approval
    approvers: ["ops-lead-pubkey", "eng-lead-pubkey"]
    timeout: 30m

  - id: trigger-deploy
    action: webhook
    url: "https://ci.internal/deploy"
    method: POST
    body: '{"version": "{{vars.release_tag}}"}'

Every step emits events into the relay's event log. Search queries across workflow execution history the same way they search channel messages.

Supported Action Types

send_message — post to a channel
send_dm — send a direct message
add_reaction — add an emoji reaction
webhook — trigger an external HTTP request
wait_delay — pause for a specified duration
wait_approval — wait for human approval (partially implemented)

Technical Architecture

Backend (Rust)

Buzz's relay is a Rust workspace with focused crates:

buzz-relay          Axum WebSocket + REST HTTP server
buzz-store          Postgres event storage + full-text search
buzz-pubsub         Redis pub/sub (real-time message fan-out)
buzz-media          S3/MinIO media file storage
buzz-acp            ACP protocol harness (standalone binary)

One event log: chat messages, canvas updates, code reviews, CI events, and workflow step results are all Nostr events in the same Postgres table with the same full-text index. Searching "deploy" returns related chat discussion, canvas docs, and Git commit statuses from a single query.

Security model:

Tenant context resolved from server hostname, not client-supplied tags — agents can't forge cross-community access
Channel-scoped tokens cannot publish global events
Subscription delivery re-checks permissions at fan-out time, catching changes after subscription
Ephemeral events (typing indicators, presence) are verified and access-checked but never written to Postgres

Desktop App (Tauri + React)

# Start relay + desktop together
just dev

Tauri provides system capabilities (file access, notifications) via Rust; React handles UI rendering. The result is a native desktop app with lower memory footprint than Electron.

Mobile (Flutter)

iOS and Android clients are Flutter-based, currently in active development.

Quick Start

Local Development

git clone https://github.com/block/buzz.git
cd buzz

# Use Hermit to manage the toolchain (recommended)
. ./bin/activate-hermit

# Or install manually: Rust 1.88+, Node 24+, pnpm 10+, just

just setup   # initialize dependencies
just build   # build all components
just dev     # start relay (ws://localhost:3000) + desktop app

Docker

docker compose up -d

Includes relay, Postgres, Redis, and MinIO (media storage).

Railway One-Click Deploy

A Railway deployment template is included in the repo — one click to deploy a private Buzz instance in the cloud.

Connect an Agent

# Install buzz-cli
cargo install buzz-cli

# Set environment variables
export BUZZ_RELAY_URL='wss://your-relay.buzz.xyz'
export BUZZ_PRIVATE_KEY='nsec...'

# Add agent to a channel
buzz channels add-member --channel CHANNEL_ID --pubkey AGENT_PUBKEY --role member

# Start the ACP harness (auto-detects locally installed agents)
buzz-acp --relay wss://your-relay.buzz.xyz --key nsec...

Resources

🌟 GitHub: block/buzz
🌐 Hosted: buzz.xyz
📖 Launch post: block.xyz/inside/introducing-buzz
💬 Hacker News: news.ycombinator.com/item?id=48632977

Summary

Buzz bets everything on one design decision: AI agents should join teams as full members, not as attached tools. That decision drives every subsequent choice — Nostr (identity not controlled by any platform), cryptographic keypairs (behavior is auditable), buzz-cli's JSON output (LLM-callable), ACP protocol (unified agent integration standard).

The one-event-log design is worth noting. Chat messages, code reviews, CI notifications, and workflow execution steps all go into the same Nostr event stream with the same full-text index. An agent querying history doesn't need to know "is this a chat event or a code event" — one search surfaces all relevant context. That's an agent-native data model, not a human-designed taxonomy with an agent adapter bolted on.

Buzz is early. Some workflow actions return NotImplemented, the mobile client is in progress, and the ecosystem is just forming. But Block has done something important: it moved the agent collaboration problem from "API integration" to "platform design," then open-sourced the whole platform for the community to evolve.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.