DEV Community

Cover image for Open Source Project of the Day (#135): CodeWiki — ACL 2026 Research-Grade Codebase Documentation with Recursive Multi-Agent Architecture
WonderLab
WonderLab

Posted on

Open Source Project of the Day (#135): CodeWiki — ACL 2026 Research-Grade Codebase Documentation with Recursive Multi-Agent Architecture

#ai #codegraph #opensource #codeunderstand

Introduction

"CodeWiki documents itself."

This is article #135 in the Open Source Project of the Day series. Today's project is CodeWiki — an automated codebase documentation generation framework from FPT Software's AI4Code research team, accepted at ACL 2026 (the Association for Computational Linguistics annual conference).

Codebase documentation is a problem that's gone unsolved for decades. Function-level annotations handle "what does this function do." But cross-file, cross-module architectural understanding — why this component exists here, what data flow passes through which layers — has never had a systematic solution.

CodeWiki's approach: address the scale problem with a recursive multi-agent architecture. Tree-Sitter parses ASTs to build a dependency graph, topological sorting determines processing order, leaf modules are processed first and synthesized upward, and modules too complex for a single pass spawn sub-agents automatically. The output is complete Markdown documentation with Mermaid architecture diagrams.

What You'll Learn

  • CodeWiki's three-stage pipeline: AST parsing → recursive multi-agent generation → hierarchical synthesis
  • Dynamic Delegation: how an agent decides it can't handle a module and splits the work
  • CodeWikiBench evaluation framework: how to rigorously measure AI-generated documentation quality
  • Differences vs. DeepWiki, deepwiki-open, and OpenDeepWiki
  • Incremental update design: --update only regenerates changed modules

Prerequisites

  • Basic familiarity with ASTs (Abstract Syntax Trees)
  • Experience maintaining a codebase — understanding why documentation is hard helps
  • Some awareness of LLM multi-agent systems

Project Background

Why Codebase Documentation Is Hard

Function-level comment generation is solved. GitHub Copilot and AI coding assistants handle it. The difficulty starts at higher abstraction levels:

Function level (solved):
"This function accepts a user_id, queries the database,
 and returns a user object"

Module level (harder):
"This auth module depends on user_service and cache_layer.
 JWT verification is primary; session auth is the fallback."

Repository level (what CodeWiki targets):
"What is the overall architecture of this codebase?
 How does data flow from the API layer to storage?
 What are the dependency relationships between modules?"
Enter fullscreen mode Exit fullscreen mode

The challenge at repository level: dependencies cross files, architectural descriptions require a global view, and large codebases far exceed single LLM context windows.

Author / Team

  • Organization: FSoft-AI4Code (FPT Software's AI research team, Vietnam's largest IT company)
  • Paper: ACL 2026 Findings (aclanthology.org/2026.findings-acl.288)
  • License: MIT
  • Language: Python 3.12+

Project Stats

  • ⭐ GitHub Stars: 1,500+
  • 🍴 Forks: 218+
  • 📄 License: MIT
  • 🎓 Paper: ACL 2026

Core Architecture: Three-Stage Pipeline

Stage 1: Repository Analysis (AST + Dependency Graph) 

# CodeWiki uses Tree-Sitter to parse all source files
# Extracts: functions, classes, cross-language dependencies
# Normalizes everything to a depends_on relation
# Builds directed graph G=(V, E)

Codebase
  ↓
Tree-Sitter AST parsing (9 languages)
  ↓
Extracts: function definitions, class definitions, module imports
  ↓
Cross-file dependencies normalized to depends_on directed graph
  ↓
Topological sort → finds zero-in-degree nodes (leaf modules with no dependencies)
Enter fullscreen mode Exit fullscreen mode

The dependency graph matters because A depends_on B means understanding A requires understanding B first. Topological sort gives the processing order — dependencies before dependents.

Stage 2: Recursive Multi-Agent Documentation Generation

This is CodeWiki's central design.

The problem with naive LLM processing:

Large module → exceeds LLM context window → truncation → degraded documentation
Enter fullscreen mode Exit fullscreen mode

CodeWiki's Dynamic Delegation:

Process a module
    ↓
Assess module complexity
    ↓
    ├── Fits in single pass → generate documentation directly
    │
    └── Exceeds capacity → spawn sub-agents
                           Sub-module 1 → Sub-agent 1
                           Sub-module 2 → Sub-agent 2
                           Sub-module 3 → Sub-agent 3
                               ↓
                           All sub-modules complete → parent agent synthesizes
Enter fullscreen mode Exit fullscreen mode

Each leaf agent receives:

  • Full source code access for its module
  • Global module tree visibility (knows its place in the overall architecture)
  • Dependency graph traversal tools (can query upstream and downstream dependencies)
  • Global registry access (prevents duplicate generation — uses cross-references instead)

Stage 3: Hierarchical Synthesis (Bottom-Up) 

Leaf module documentation (bottom level, no dependencies)
        ↓
Parent module merges child docs + generates architectural summary
        ↓
Top-level overview (overall architecture + system interaction diagrams)
        ↓
Mermaid visualization generation:
    - Architecture diagrams
    - Data flow diagrams
    - Sequence diagrams
Enter fullscreen mode Exit fullscreen mode

Output structure:

./docs/
├── overview.md          ← Top-level architecture overview
├── module_A.md          ← Per-module detailed documentation
├── module_B.md
├── module_tree.json     ← Machine-readable module tree
├── metadata.json        ← Generation metadata
└── index.html           ← Generated with --github-pages flag
Enter fullscreen mode Exit fullscreen mode

Evaluation: CodeWikiBench

CodeWiki also contributed a benchmark — CodeWikiBench, designed specifically to evaluate AI-generated codebase documentation quality.

Traditional text similarity metrics (BLEU/ROUGE) don't work for documentation quality. A technically correct but verbose document scores high; a concise and precise document might score lower. Neither tells you whether the documentation is actually useful.

CodeWikiBench's evaluation approach:

1. Extract hierarchical evaluation rubrics from official project documentation
   Generated by multiple models (Claude Sonnet 4, Gemini 2.5 Pro, Kimi K2)
   Semantic reliability: 73.65%, structural reliability: 70.84%

2. Multiple Judge Agents make binary pass/fail decisions
   Evaluated only at leaf nodes — avoids fuzzy intermediate scoring
   Judge models: Gemini 2.5 Flash, GPT OSS 120B, Kimi K2

3. Weighted scores aggregated upward from leaves
   With standard deviation confidence bounds
Enter fullscreen mode Exit fullscreen mode

Key results:

System Average Score
OpenDeepWiki (open-source) 47.13%
deepwiki-open (open-source) 50.05%
DeepWiki (closed-source, Cognition AI) 64.06%
CodeWiki 68.79%

CodeWiki leads clearly on Python, JavaScript, and TypeScript (TypeScript +18.54%, Python +9.41%). Both CodeWiki and DeepWiki perform worse on C and C++ — the paper attributes this to "language-specific parsing complexity" rather than repository size.

Quick Start

Installation 

git clone https://github.com/FSoft-AI4Code/CodeWiki.git
cd CodeWiki
pip install -e .
Enter fullscreen mode Exit fullscreen mode

Generating Documentation 

# Basic: generate docs for current directory
codewiki run . --output docs/

# Specify LLM provider
codewiki run . --provider openai --model gpt-4o

# Use Claude
codewiki run . --provider anthropic --model claude-opus-4-6

# Use Claude Code subscription (no API key needed)
codewiki run . --provider claude-code

# Generate GitHub Pages
codewiki run . --github-pages

# Incremental update (only regenerate modules changed since last run)
codewiki run . --update
Enter fullscreen mode Exit fullscreen mode

Supported LLM Providers

Provider Auth
OpenAI API Key
Anthropic Claude API Key
Azure OpenAI API Key
AWS Bedrock IAM
Atlas Cloud API Key
Claude Code Subscription — no API key
Codex CLI Subscription — no API key

Comparison with Similar Open-Source Projects

Several projects occupy this space:

deepwiki-open (AsyncFuncAI)

  • ⭐ 17,100+ Stars
  • Open-source re-implementation of Cognition AI's DeepWiki product
  • Python + TypeScript, supports GitHub/GitLab/Bitbucket
  • Simpler deployment, Web UI included
  • CodeWikiBench score: 50.05% (vs CodeWiki's 68.79%)
  • Best for: quick deployment, teams that want a web interface

OpenDeepWiki (AIDotNet)

  • C# implementation (.NET ecosystem)
  • Also a DeepWiki re-implementation, targeting .NET developers
  • CodeWikiBench score: 47.13%
  • Best for: .NET / Windows enterprise environments

context-labs/autodoc

  • Early experimental project (2023), GPT-4/Alpaca based
  • LlamaIndex-style codebase indexing approach
  • Less actively maintained but established the foundational design patterns for this category

Summary Table

Project Stars Quality Deployment Stack Best For
CodeWiki 1.5k Highest (68.79%) CLI Python Large codebases, quality-first
deepwiki-open 17.1k Mid (50.05%) Web UI Python/TS Fast deployment, web interface
OpenDeepWiki — Mid (47.13%) Web UI C# .NET environments
autodoc Low activity Early CLI Node.js Reference only

Links and Resources

Conclusion

CodeWiki contributes on two levels: a usable tool and an evaluation benchmark.

On the tool side, Dynamic Delegation solves a real engineering problem. Large codebases can't fit in a single LLM context window. Hierarchical recursive synthesis maintains documentation quality across 86K to 1.4M line codebases instead of degrading at the boundary. The benchmark numbers support that claim.

On the evaluation side, CodeWikiBench fills a gap. There was no rigorous, purpose-built framework for measuring codebase documentation quality before this work. That contribution stands independently of the CodeWiki tool itself.

The limitations are real. C and C++ performance trails Python and TypeScript. Stars (1.5k) lag far behind deepwiki-open (17.1k), which signals a gap in usability and community reach that quality benchmarks don't close on their own.

For engineers who need to document large codebases and care about output quality, CodeWiki has the most credible benchmark numbers among open-source options. For quick deployment with a web interface, deepwiki-open is the lower-friction choice.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

Top comments (0)