DEV Community: WonderLab

LLM-Driven Automated Testing Series (02): The Unified Technical Foundation — Visual Grounding / DOM Semantics / Computer Use

WonderLab — Tue, 22 Sep 2026 01:21:00 +0000

Same problem, three answers

"Find the submit button on this page and click it."

Hand that instruction to three different open-source GUI agent projects and you'll get three entirely different implementations:

One feeds the full screenshot to a specially-trained visual grounding model, which directly outputs a coordinate like (842, 613)
Another parses the page's DOM or accessibility tree, finds the node with role=button, name="Submit", and calls its click handler directly
A third skips any dedicated locating step entirely — a general-purpose multimodal model just looks at the screenshot and decides "click here" on its own

These three paths aren't interchangeable implementation details — they're distinct architectural decisions, each with clear trade-offs and boundaries. This post builds the vocabulary the next 12 case-study posts will reuse. Without it, every post would have to re-explain "which locating approach this project uses" from scratch.

Path 1: Pixel-Based Visual Grounding Models

How it works

This path treats the GUI as an image. Given a natural-language reference ("the submit button") and a screenshot, the model directly outputs the element's coordinates or bounding box on screen.

This is a dedicated sub-task in the literature called GUI grounding: it doesn't decide "what to do next" — it only maps a natural-language reference precisely onto screen coordinates. Representative open-source models include UGround, Aria-UI, and Alibaba's self-trained GUI-Owl (used in Mobile-Agent-v3, covered in a later post).

The typical architecture is two-stage:

User task: "Submit this form"
        ↓
Planning model (can be a general LLM): "Next step is to click the submit button"
        ↓
Visual grounding model (dedicated, e.g. UGround): "submit button" + screenshot → (842, 613)
        ↓
Execution layer: click at (842, 613)

Planning and grounding are two separate models, each specialized for one job.

Trade-offs

Advantages: no dependency on any application-internal structured data — pure "looking," which makes it naturally cross-platform (web, mobile, desktop, even game UIs — anywhere you can take a screenshot). A key result from the UGround paper: this path beats accessibility-tree-dependent approaches (Path 2 below) by up to 20 absolute percentage points on multiple GUI grounding tasks, because real-world accessibility trees are often incomplete or unreliable.

Costs:

Every locating step requires a visual model inference pass — slower and more expensive than reading structured data directly
Accuracy drops sharply for small elements and dense layouts (the ScreenSpot-Pro numbers below quantify how much)
The grounding model itself needs dedicated training and maintenance — you can't get high accuracy "for free" from a general-purpose model

Path 2: DOM / Accessibility-Tree Structured Understanding

How it works

This path doesn't look at pixels — it looks at structure. Web pages have a DOM tree; desktop and mobile apps have an accessibility tree (the semantic view the OS exposes for screen readers and other assistive tools). Serialize that tree into text and hand it to an LLM directly: "Given this structure, which node is the submit button?"

Once the target node is found, subsequent actions don't need coordinates at all — you call the node's exposed click(), setValue(), etc. directly, the same mechanism traditional Selenium/Playwright scripts already use. The only thing that changes is that "finding the node" moves from a hardcoded selector to LLM-based semantic matching.

Trade-offs

Advantages:

Structured data is text, so token cost is far lower than images, and it's faster
Locating accuracy doesn't degrade with screen resolution or element size — if the node is in the tree, semantic matching is precise regardless
Can reuse an existing automation framework's execution layer (Playwright, Appium) — the change is isolated to the "locating" step

Costs:

Strongly dependent on the app exposing complete, reliable structured information. Canvas-rendered content, embedded WebViews, and heavily custom-rendered mobile widgets often show up as an opaque black-box node in the accessibility tree with no semantic information at all — at that point this path simply fails and must fall back to a visual approach
Real-world DOM/accessibility trees can be enormous (modern SPAs routinely have thousands of nodes); stuffing the whole thing into an LLM's context is expensive, requiring pruning or downsampling — this is exactly the problem work like "DOM Downsampling for LLM-Based Web Agents" sets out to solve
Tree structures differ significantly across platforms (Web DOM, Android View tree, iOS UIKit tree), so one semantic-understanding pipeline doesn't transfer cleanly across platforms

Path 3: Computer Use — Raw Coordinate Clicking

How it works

This is the most "brute-force," yet also the most commercially deployed path of the past two years: Anthropic's Computer Use and OpenAI's Operator/CUA both fall into this category.

The key difference from Path 1: there's no separate "grounding model" step. A general-purpose multimodal model takes the screenshot directly as input and directly outputs the next action — "click at (842, 613)," "type text," "press Tab" — perception and decision-making happen in a single pass by the same model. There's no "planning model + dedicated grounding model" division of labor.

Screenshot → general multimodal model (perception + decision combined) → directly outputs: click(842, 613)

Trade-offs

Advantages:

Simplest architecture — no separate grounding model to maintain, no platform-specific structured data to parse
Because it has zero dependency on any interface the app exposes, it has, in principle, the highest ceiling on generality — "if you can screenshot it, you can operate it" — and it's the technical basis for turning an agent from "a web automation tool" into "a general assistant that operates a computer"

Costs:

Locating accuracy is the weakest of the three paths on public benchmarks — the numbers in the next section quantify this
Highest latency and cost: every single action requires a full multimodal inference pass, and tasks often need many consecutive steps to complete — this is the most expensive path of the three to run
No structured-data fallback: Path 2 can approach near-100% precision when the structure is reliable; Path 3 has no such ceiling guarantee — it's purely bounded by the model's visual grounding ability

Quantifying the gap with benchmarks

Saying "one path is better" without numbers is meaningless. A few public benchmarks let us compare the three paths on the same scale.

The ScreenSpot family: testing "locating" in isolation

ScreenSpot was the earliest GUI grounding benchmark: given a natural-language reference and a screenshot, the model must point to the target element's coordinates. Leading models already score above 90% accuracy on this benchmark.

ScreenSpot-Pro is the hardened version: all screenshots come from real professional desktop software (CAD tools, IDEs, data-analysis apps), at higher resolution with smaller, denser interface elements. The same leading models drop to 37.1% (one of the highest scores reported by a 2025-generation model — most models score far lower).

Benchmark          What it measures              Leading model accuracy
─────────────────────────────────────────────────────────────────────
ScreenSpot          Locating in simple UIs         90%+
ScreenSpot-Pro       Locating in professional UIs   ~37% (best case)

That cliff-edge drop reveals a key fact: "locating accuracy" isn't a fixed property of a model — it collapses sharply as interface density and element size push in the wrong direction. This is exactly why mobile automation (covered in five dedicated posts, 05-09) leans so heavily on dedicated grounding models — icons on a phone screen tend to be smaller and denser than buttons on desktop software.

OSWorld: testing end-to-end task success

ScreenSpot only tests "can you find the element." OSWorld tests whether a full task actually gets completed — executing a multi-step task in a real OS environment (e.g., "clean up this data in LibreOffice Calc and export it as a PDF"), which requires planning, locating, execution, and error recovery across the whole chain.

The OSWorld paper reports a human baseline of 72.36% task completion, while early baseline models (e.g. GPT-4V) achieved only 12.24% — a gap of nearly 60 percentage points. The paper explicitly attributes the bottleneck to GUI grounding and operational knowledge, not task comprehension itself. That number has since closed rapidly as dedicated agent architectures (e.g. the Agent S series) improved — by 2025-2026, leading approaches reach roughly 65%-70%+, closing in on the human baseline, but that gain came from agent-architecture-level engineering — planning, reflection, retry loops — not purely from a better underlying grounding model.

                     Human baseline    Early baseline model    2026 leading approaches
──────────────────────────────────────────────────────────────────────────────────
OSWorld task success   72.36%           12.24%                  ~65%-70%

The takeaway: grounding accuracy and task success rate are two different things. A perfectly accurate grounding model still won't get you a high end-to-end success rate if the agent can't re-plan after a failure or verify whether its action achieved the intended effect. That's why "planning + reflection mechanisms" will be just as important a topic as "which grounding model was chosen" when we get to mobile automation projects like ARTEMIS and Mobile-Agent-v3.

The three paths aren't mutually exclusive

In practice, mature open-source projects rarely bet purely on one path. The common pattern is layered fallback: try Path 2 first (DOM/accessibility tree — fast and precise), and fall back to Path 1 or Path 3 (visual grounding) whenever reliable structured data isn't available. AppAgent, covered in a later post (08), is a textbook example of this "parser tree + visual features, dual input" design — the two paths aren't a mutually exclusive technology choice, but two tiers of fallback ranked by reliability within a single system.

Concept vocabulary (reused across the series)

Term	Meaning	Referenced in
GUI Grounding	Mapping a natural-language reference to on-screen coordinates/elements	04-09
Visual grounding model	A model specifically trained for GUI grounding (UGround, GUI-Owl)	05, 07
Accessibility tree / DOM semantic understanding	Locating elements via structured views instead of pixels	04, 08, 10
Computer Use	A general multimodal model directly outputting coordinate actions, no separate grounding stage	09
ScreenSpot / ScreenSpot-Pro	GUI grounding benchmarks that measure single-step locating accuracy only	05-09
OSWorld / AndroidWorld	End-to-end benchmarks measuring full multi-step task success rate	05-09

Summary

There are three technical paths for GUI element locating: dedicated visual grounding models (broadly cross-platform, but slow and expensive), DOM/accessibility-tree structured understanding (fast and precise, but dependent on the platform exposing reliable structured data), and Computer Use raw coordinate clicking (simplest architecture, highest generality ceiling, but the lowest locating-precision ceiling)
The ScreenSpot → ScreenSpot-Pro cliff (90%+ dropping to ~37%) shows that locating accuracy collapses sharply with interface density — it isn't a fixed model capability
OSWorld's human-vs-model gap (72.36% vs. an early baseline of 12.24%) shows that end-to-end task success depends on more than grounding precision alone — planning and error-recovery mechanisms matter just as much
Mature systems typically use layered fallback rather than betting on a single path — this vocabulary will recur throughout every case study that follows

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 189): NanoJev — A 0.6B Parallel Decision Model That Outputs Probability Distributions Instead of Generating Tokens

WonderLab — Tue, 22 Sep 2026 01:19:41 +0000

Introduction

"A decision doesn't need to write a sentence to explain itself — it just needs a probability distribution."

This is the 189th article in the "One Open Source Project a Day" series. Today's project is NanoJev.

Most LLM-driven decision-making approaches can't escape an awkward efficiency problem: the model autoregressively generates a chunk of text (something like "I think we should choose option B, because..."), and the final decision has to be parsed out of that text afterward. This requires token-by-token decoding — slow, costly, and the "confidence" the model expresses is usually just an adjective it happened to use, not a genuinely calibrated probability.

If a task's essence is really "given a state and a set of candidates, tell me the probability of each candidate," why make the model generate a wall of text first and parse it after? NanoJev's answer: skip the generation, output the distribution directly. It's a lightweight recreation of a parallel decision model system called "Jev." Built on a Qwen3-0.6B backbone with dedicated decision heads, it takes a state and a question as input and, in a single forward pass, outputs a complete probability distribution — zero output-token decoding.

954 Stars, MIT License, validated with maze navigation and Snake game benchmarks.

What You Will Learn

How NanoJev structures a decision as a "state + question + candidate set" triple
Three decision head types: dynamic Choice, Boolean judgment, and Ordered Score
Why "zero output-token decoding" fits decision-making better than autoregressive generation
The full training pipeline: build queries → organize data → train → evaluate → serve
How NanoJev compares against original Jev and an untuned Qwen3-0.6B on game benchmarks

Prerequisites

Basic understanding of how LLMs run inference (autoregressive generation vs. a single forward pass)
Familiarity with basic probability and classification/regression concepts
Optional: familiarity with reward modeling concepts in reinforcement learning

Project Background

What It Is

NanoJev's official positioning: "A 0.6B parallel decision model. States and questions in, complete probability distributions out — with zero output-token decoding."

Worth clarifying up front: this is not an image generation or editing tool. It's a structured decision model/reasoning framework — essentially it keeps an LLM's language understanding ability while re-engineering the output side from "generate text" to "output a calibrated probability distribution," aimed at use cases like game agent decision-making and probability calibration research.

Team and Background

Author: TianyuCodings
License: MIT License
Base model: Qwen3-0.6B
Related resources: model repo C-Tianyu/NanoJev, dataset C-Tianyu/NanoJev-Data (both on HuggingFace)

Project Stats

⭐ GitHub Stars: 954
🍴 Forks: 127
👀 Watchers: 8
📄 License: MIT
📊 Commits: 13

What It Does

The Problem It Solves

Traditional LLM decision-making approach:
  State + question → autoregressively generate text ("I think we should
  pick B, because...")
  ↓ parse the text to extract the final decision
  ↑ token-by-token decoding is slow and costly
  ↑ "confidence" is just an adjective in the text, never truly calibrated

NanoJev's approach:
  State + question + candidate set → a single forward pass →
  directly output a complete probability distribution
  ↑ zero output-token decoding, fast
  ↑ the distribution is trained and calibrated, usable directly for
    ranking / greedy selection / probability sampling
  ↑ a single forward pass can batch multiple independent states and questions
     (official example: "6 states · 18 questions · 44 candidate paths ·
     1 backbone forward")

Use Cases

Game agent decision-making
- Maze navigation (8×8 up to 50×50 in various sizes), real-time decisions for a Snake game agent
Local safety judgment
- Fast boolean judgments for scenarios like collision avoidance, combined with code-based planning
Probability calibration research
- Calibration experiments using CE (cross-entropy)/Brier loss and paired proper-reward learning
Scenarios needing batched decisions
- Evaluate multiple candidate paths in a single forward pass, suited to applications needing high-throughput decision-making

Quick Start

Simplest path: run the interactive demo locally (no model weights needed)

git clone https://github.com/TianyuCodings/NanoJev.git
cd NanoJev
python3 -m http.server 8080 --bind 127.0.0.1 --directory web

Visit http://127.0.0.1:8080/side-by-side.html for a three-panel comparison demo, or arcade.html (dark arena demo) and comparison.html (early benchmark viewer).

Full run: download model weights and start the service

# Install dependencies
python -m pip install -r requirements-toy.txt

# Download model checkpoint and dataset
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='C-Tianyu/NanoJev', local_dir='checkpoints/NanoJev',
    allow_patterns=['best.safetensors', 'config.json', 'tokenizer/*', 'backbone_config/*'],
)
snapshot_download(
    repo_id='C-Tianyu/NanoJev-Data', repo_type='dataset', local_dir='data/NanoJev',
)
"

# Start the persistent service
python scripts/serve_decisions.py \
  --checkpoint-dir checkpoints/NanoJev \
  --web-root web --port 8765

Once started, the service loads the model once and accepts repeated batches of decision requests via POST /api/evaluate.

Core Features

1. Three Decision Head Types

Type	Range	Mechanism
Dynamic Choice	2–255 candidates	A shared scalar head + set attention, outputs a probability for each candidate
Boolean	A single proposition	Single-path sigmoid, outputs the probability the proposition is true
Ordered Score	2–10 levels	Computes a probability-weighted expectation over the ordered level distribution

2. Triple-Structured Decisions

Every decision is defined by a state, a question, and a candidate set. A shared decision head processes the input and returns the corresponding distribution output for the candidate set based on the decision type.

3. Five-Step Training Pipeline

Build queries
  ↓ generate states, questions, candidate descriptions, and target distributions
Organize data
  ↓ keep related maps, rules, and variants in the same data split
Train
  ↓ Qwen3-0.6B init + decision-head warmup + complete-question distribution losses
Evaluate
  ↓ measure probability quality + game controller records actual actions
Serve and visualize
  ↓ reuse a persistent model endpoint, replay full trajectories in the browser

4. Multiple Pretrained Checkpoint Variants

Variant Name	Use
`variants/local_atomic_seed17`	50×50 maze demo
`variants/games_gold_seed17`	Snake demo
`variants/games_api_seed17`	Full map comparison benchmark
`variants/events_ce_seed17` / `events_brier_seed17` / `events_paired_seed17`	Calibrated decision experiments (different loss functions)

A Deeper Look

The Performance Implications of "Zero Output-Token Decoding"

NanoJev's core architectural decision is to abandon autoregressive generation entirely, re-engineering the LLM's output side into structured decision heads. The official documentation gives a fairly intuitive number for the performance difference this brings:

"6 states · 18 questions · 44 candidate paths · 1 backbone forward"

Meaning:
  6 independent states + 18 questions + 44 candidate paths
  ↓ all folded into a single backbone forward pass
  ↓ instead of 44 separate autoregressive generations

The cost of autoregressive generation accumulates token by token — generating a chunk of explanatory text might require tens to hundreds of forward passes (one per token). NanoJev compresses the evaluation of 44 candidate paths into "1 backbone forward pass" — essentially swapping "expressing confidence in text" for "expressing probability directly through the model's internal numerical output," bypassing the entire autoregressive decoding cost chain.

The Design Trade-offs Behind Choice / Boolean / Score Heads

Why not use one generic head for every decision type? The three head types actually correspond to three fundamentally different statistical problems:

Choice (dynamic selection):
  Variable candidate count (2-255) → needs "set attention"
  ↑ the model must understand "these candidates form a mutually
    exclusive set," not independent judgments

Boolean:
  A single proposition, binary yes/no → a single-path sigmoid is enough
  ↑ no need to be aware of other candidates

Score (ordered evaluation):
  Levels have an order relationship (1 star to 5 stars) → can't be
  treated as a plain classification problem
  ↑ using a probability-weighted expectation lets a non-integer
    expectation like "3.5 stars" emerge naturally

This design reflects a simple but easily overlooked principle: when decisions have different statistical structures, the loss function and output head should differ too. Forcing "which option," "true or false," and "what rating" into the same softmax loses the structural information specific to each question type.

Real-World Training Results: Small Model vs. Fine-Tuning

NanoJev was benchmarked against original Jev and an untuned Qwen3-0.6B across three game benchmarks, and the results are fairly compelling:

Benchmark	NanoJev	Jev (original)	Untuned Qwen3-0.6B
50×50 maze (attempts/collisions)	244 attempts / 36 collisions, goal reached	2,738 attempts / 1,044 collisions, goal reached	—
12×12 Snake (food eaten/alive)	27 food, alive at horizon	30 food, alive at horizon	25 food, trapped
40-map navigation benchmark (test/OOD)	95% / 90%	100% / 95%	35% / 15%

A few points worth noting:

The untuned Qwen3-0.6B lags significantly behind (35%/15% vs. NanoJev's 95%/90%), showing that the purpose-trained decision heads deliver a real improvement — not just a rebranded model
NanoJev requires far fewer attempts than original Jev on the 50×50 maze (244 vs. 2,738), suggesting the lightweight recreation is actually more efficient on this specific task, likely due to more focused training data
NanoJev's overall accuracy is slightly lower than original Jev (e.g., 95% vs. 100% on the navigation benchmark), which is expected — recreating a system with fewer resources involves a reasonable trade-off in performance

Positioning: A Research Recreation, Not a Commercial Product

NanoJev explicitly positions itself as a "lightweight, reproducible" research alternative. That positioning is clear: it doesn't aim to outperform original Jev, but rather to let more researchers validate and extend the "parallel decision model" idea at a smaller training cost with a simpler deployment path. Roadmap items like expanding data scale and RLCD (contrastive distillation reinforcement learning) expansion all point toward "continuing to explore the boundaries of this architecture" rather than "polishing it into a product."

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/TianyuCodings/NanoJev
🤗 Model: C-Tianyu/NanoJev (HuggingFace)
🤗 Dataset: C-Tianyu/NanoJev-Data (HuggingFace)
📄 License: MIT License
🐛 Issues: GitHub Issues

Related Resources

Qwen3 — The 0.6B backbone network NanoJev is built on
Brier Score — One of the calibration scoring methods used in NanoJev's training

Summary

Key Takeaways

Zero output-token decoding: abandons autoregressive generation, using decision heads to output probability distributions directly, sidestepping the performance overhead of token-by-token decoding
Triple-structured decisions: state + question + candidate set clearly bounds every decision query
Three decision heads, each fit for purpose: Choice/Boolean/Score correspond to distinct statistical problem structures rather than being forced into a single softmax
Real benchmark validation: significant improvement over untuned Qwen3-0.6B on maze navigation and Snake game benchmarks, with a reasonable performance trade-off versus original Jev
Research-oriented positioning: explicitly a lightweight recreation rather than a commercial product, serving further exploration of the "parallel decision model" direction

Who This Is For

Game AI / agent decision-making researchers: need a lightweight, reproducible baseline for parallel decision models
Researchers focused on LLM efficiency: interested in architectural changes that skip autoregressive generation in favor of directly outputting structured distributions
Probability calibration researchers: want to study the practical effect of CE/Brier loss and paired reward learning in decision-making scenarios
Teaching/reproduction-oriented learning contexts: want to understand a "decision head + backbone" architecture without training a large-scale system from scratch

One-Line Verdict

NanoJev raises a question worth sitting with: if a task's essence is producing a probability distribution, why detour through generating a paragraph of text at all?

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

LLM-Driven Automated Testing Series (01): Why Testing Is LLM's New Battlefield

WonderLab — Mon, 21 Sep 2026 03:06:25 +0000

A scenario every test engineer has lived through

A product manager moves a button by a few pixels — barely noticeable to the eye. The next day, 40 tests fail red in CI simultaneously.

After half an hour of digging, the culprit isn't broken business logic. It's a hardcoded CSS selector, #submit-btn-v2, that can no longer find the element — the DOM structure changed and the id changed with it. The business code is entirely correct; the test script died first.

This is one of the most common, most labor-draining, and least technically interesting failure modes in automated testing. It's also the starting point of this series: can LLMs turn this problem — and a few other long-standing ones in the testing world — from "brute-force human maintenance" into something that's actually engineered?

Three long-standing problems in traditional automated testing

Before discussing what LLMs can do, let's name the problems clearly. None of these are new — the industry has debated them for over a decade without a cheap, general solution.

Problem 1: Fragile Locators

Automated test scripts have to "find" an element on the page before they can act on it — click a button, fill a form, read some text. The common ways to locate an element are CSS selectors, XPath, element IDs, or raw coordinates.

All of them share the same fatal assumption: the UI structure won't change after the test script is written. In reality, UI structure changes constantly — redesigns, A/B tests, a front-end framework rewrite that restructures the component tree. The more specific a script's dependency on UI structure, the more fragile it becomes.

Selector specificity     Robustness     Maintenance cost
────────────────────────────────────────────────────────
#submit-btn-v2           Very low       Breaks on any UI tweak
.btn.primary              Low            Breaks on class refactor
//div[3]/button[1]        Very low       Breaks on DOM hierarchy change
Coordinates (320, 480)    Very low       Breaks on resolution/layout change

The industry has a name for this: "selector hell." The larger a test team grows, the higher the proportion of engineering time spent maintaining locators — not writing new tests, but fixing the locating logic of old ones.

Problem 2: The Oracle Problem (how do you know if the result is right?)

"Oracle" in testing refers to the standard used to judge whether an output is correct. A traditional unit test's oracle is explicit: assert result == 42. But in many testing scenarios, the oracle itself is fuzzy:

A screenshot differs from the last version by a few pixels — is that a bug,
  or just font-rendering jitter?
The returned JSON has an extra field — is that an API change,
  or dirty test-environment data?
The user flow completed, but the page "just doesn't look right" —
  hard to say exactly what's wrong

The traditional fix is to make the oracle explicit: pixel-level diffing (what threshold counts as a real change), schema validation (which fields must exist), assertion chains (explicitly checking state at each step). The shared problem with these approaches: the stricter the oracle, the more false positives (flagging meaningless changes as failures); the looser the oracle, the more false negatives (real bugs slip through).

Problem 3: Maintenance Cost (test scripts age faster than the business code)

This is a consequence of the first two problems, but it deserves its own callout because it's the direct reason teams eventually abandon automated testing.

A real pattern: teams that iterate faster on features tend to end up with lower actual test coverage over time — not because nobody wants to write tests, but because tests go stale the moment they're written, nobody has time to fix them, and eventually they get skipped or ignored outright. The entropy of the test codebase grows faster than the team's capacity to maintain it.

These three problems reinforce each other: fragile locators → tests fail red constantly → maintenance cost rises → the team has no time to maintain them → coverage gets cut → the oracle problem becomes harder to catch (because coverage itself is shrinking).

Why now: testing happens to be a "clear right-or-wrong" scenario

For the past two years, LLM Agent adoption has stalled in the same place across many domains: how do you know the Agent got it right? Writing code, making decisions, generating content — the "correctness" of these outputs is usually continuous and subjective, requiring a whole separate evaluation layer to judge (which is exactly what the eval-series on this blog is about).

Testing is the exception. A testing task carries its own explicit success signal:

Unit test passes / fails             —— boolean, no middle ground
UI element found / not found         —— boolean
Assertion holds / doesn't hold       —— boolean
Coverage increased by X percentage points —— a quantifiable number

This means the quality of an LLM's output in a testing context can be verified using the test's own execution result, without needing an extra meta-layer of "is this LLM output reasonable." This is a structural advantage testing has over other LLM use cases — and it's why the past two years have produced a wave of open-source projects that actually work in production, not just proof-of-concept demos.

That said, "clear right-or-wrong" only applies to the test's final execution result. Whether a test script should be generated, whether a locator should self-heal, whether a failure is a real bug or an environment flake — these judgments remain fuzzy. That's exactly where LLMs have room to intervene, not in the final "did the test pass" verdict itself.

Three places where LLMs can plug in

Coming back to the three problems from the opening: here's where LLMs can plug in for each.

Fragile locators   → Semantic locating ("find the submit button" instead of "find #submit-btn-v2")
                    → Self-healing locators (when the UI changes, re-find the element
                      semantically instead of the script just dying)

Oracle problem      → Semantic judgment ("is this visual change an actual bug to the user")
                    → instead of pure pixel/structural comparison

Maintenance cost    → Coverage-driven automatic test generation
                    → Automatic failure classification (real bug / environment issue / test issue)

These three threads map to what the rest of the series will cover: unit test generation (03), Web/mobile UI automation (04-09), self-healing locators (10), API testing (11), visual regression (12), and failure classification (13).

The next post (02) will first lay out the shared technical foundation behind all of these approaches — visual grounding, DOM-based semantic understanding, and pure coordinate-clicking Computer Use. Each has its own principles and trade-offs, and this vocabulary will be reused in every case study that follows.

What this series won't do

Setting expectations upfront:

This is not a trend-report on "AI testing." Each post picks one technical point paired with a verifiable open-source project — no "reportedly," no "may become possible in the future" claims that can't be checked
This isn't about evaluating LLM output quality. That's the eval-series's territory. This series is about "using LLMs to do testing," not "evaluating LLM outputs"
Limitations won't be glossed over. Every technical approach comes with a discussion of where it breaks down and what it costs

Summary

Fragile locators, the oracle problem, and maintenance cost are the three long-standing problems in automated testing, and they reinforce each other
Testing tasks carry an explicit success signal (pass/fail) — this is testing's structural advantage over other LLM use cases
LLMs plug in at semantic locating/self-healing, semantic oracle judgment, and automatic generation/classification — not at replacing the final "did the test pass" verdict itself

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 188): ARTEMIS — Google's Open-Source Mobile AI Automation Framework, Let AI Assistants Operate Phones Like a Human

WonderLab — Mon, 21 Sep 2026 02:01:12 +0000

Introduction

"A test script breaks the moment a button moves — an AI that actually understands the interface doesn't."

This is the 188th article in the "One Open Source Project a Day" series. Today's project is ARTEMIS.

Mobile UI automation testing has long lived with a fundamental tension: traditional scripts based on element IDs or coordinates are fast, but extremely brittle — a small layout tweak breaks test cases en masse, and maintenance cost grows linearly with how fast the app iterates. Vision-model-driven approaches are more robust, but slow and expensive, making them a poor fit for large-scale regression testing.

ARTEMIS is Google's attempt to find a balance between these two extremes. It turns natural-language instructions — like "open Settings, find the Battery option, tell me the current level" — into reliable Android automation, while offering two execution modes: one optimized for speed on routine deterministic tasks, another optimized for accuracy and verifiability on complex, long-running ones. More notably, it plugs directly into Claude Code, Antigravity, Codex, and other AI IDEs via MCP, giving AI coding assistants the ability to operate real phones.

8.1k Stars, Apache-2.0, built in Python, achieving 99%+ task completion on the AndroidWorld benchmark.

What You Will Learn

How ARTEMIS turns natural-language instructions into Android automation
The architectural differences between the Flash and Pro execution modes, and when to use each
Multi-modal target localization: combining the accessibility hierarchy, OCR, and vision models
How it integrates with Claude Code and other AI IDEs via MCP
How the Python SDK plugs into existing automated testing frameworks

Prerequisites

Basic understanding of Android automation testing concepts (ADB, Accessibility services)
Familiarity with Python async programming (asyncio)
Optional: basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

ARTEMIS's official positioning: "ARTEMIS turns natural-language instructions into reliable Android automation" — letting AI assistants and test suites operate real phones like a human would.

One clarification worth making up front: this is not a security testing or fuzzing tool. It's an intelligent agent system focused on mobile UI automation testing and everyday task execution, squarely in the AI-driven mobile testing/automation framework space.

Team and Background

Organization: Google
License: Apache License 2.0
Primary language: Python (requires Python 3.12+)
Code provenance: the project notes it includes source code developed by Minitap, Inc., building on the open-source mobile-use project

Project Stats

⭐ GitHub Stars: 8,100+
🍴 Forks: 770+
👀 Watchers: 71
📄 License: Apache-2.0
🏆 Benchmark: 99%+ task completion on AndroidWorld (covering 20+ apps, 100+ multi-step tasks)

What It Does

The Problem It Solves

Traditional element-ID/coordinate-based automation scripts:
  UI layout shifts slightly → element lookup fails → tests break en masse
  ↑ Fast, but extremely brittle, high maintenance cost

Pure vision-model-driven approaches:
  Every step relies on a vision model to read the screen → accurate but slow
  ↑ Good robustness, but slow and expensive, hard to scale

ARTEMIS's approach:
  Natural-language instruction → multi-modal localization (element index first,
  vision/coordinates as fallback)
  ↓ Flash mode: routine deterministic tasks, 3-5 sec/step, fast reactive loop
  ↓ Pro mode: complex long-running tasks, 15-40 sec/step, plan-execute-verify workflow
  ↑ Choose the right mode for task complexity, balancing speed and reliability

Use Cases

Mobile app end-to-end test automation
- Describe test steps in natural language instead of maintaining brittle element-locator scripts
Bug reproduction and diagnosis
- Combined with Logcat log and screenshot capture, quickly reproduce and pinpoint issues
Exploratory stability testing
- Pro mode supports long-running, continuously monitored exploratory testing
CI/CD-integrated automated testing
- Integrate via the Python SDK into pytest and similar frameworks, folded into continuous integration pipelines
AI IDE-driven real device operation
- Let Claude Code, Antigravity, and other AI coding assistants directly operate a phone to verify functionality

Quick Start

Clone and one-command start:

git clone https://github.com/google/artemis.git && cd artemis

# macOS/Linux one-command start (auto-detects and installs ADB, scrcpy, FFmpeg, etc.)
./start.sh

Run a task directly via CLI:

uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash

Install MCP integration for an IDE:

# Install for Antigravity
uv run artemis mcp --install antigravity

# Install for all supported IDEs (including Codex)
uv run artemis mcp --install all

On first task execution, ARTEMIS installs an "Artemis Accessibility Helper" service on the device — it only reads the screen layout locally and uploads no data, and can be pre-installed, checked, or uninstalled via command.

Core Features

1. Four Ways to Use It

Method	Best For
Web visual test console (`artemis ui`)	Interactive debugging and results review
MCP server	Connecting an AI IDE to drive a real device
Developer CLI (`artemis run`)	Automated testing or benchmark scripts
Python SDK	Integrating into existing test frameworks like pytest / CI pipelines

2. Dual Execution Modes: Flash and Pro

Dimension	Flash Mode	Pro Mode
Response speed	~3-5 sec/step	~15-40 sec/step
Architecture	Single-model observe-think-act reactive loop	Multi-agent graph (Planner+Operator+Checker)
Task planning	No task plan	Planner maintains a Markdown task plan with milestones
Safety mechanism	No pre-execution safety net	Every action XML-verified first, pixel fallback
Verification	No checkpoint verification or final report	Checker verifies plan checkpoints, supports a final review before exit
Toolset	No ADB shell	Full toolset: Explorer, notes, history lookback, video analysis
Best for	Routine deterministic UI tasks	100+ step long-running workflows, continuous monitoring

3. Python SDK Integration Example

uv add "artemis-client @ git+https://github.com/google/artemis.git#subdirectory=packages/artemis-client"

import asyncio
from artemis_client import ArtemisClient


async def main():
    client = ArtemisClient(
        "http://artemis-host:8000",
        device_serial="emulator-5554",  # optional: target specific device serial
        default_profile="flash",  # "flash" (fast reactive) or "pro" (deep reasoning)
    )

    result = await client.run(
        "Open System Settings, go to 'Battery', verify battery percentage is displayed, and check for any crash dialogs.",
    )

    assert result.succeeded, f"Test failed: {result.error or result.status}"
    print(f"✅ Test Passed! Device: {result.device_serial} | Trace ID: {result.trace_id}")


if __name__ == "__main__":
    asyncio.run(main())

The SDK claims "zero runtime dependencies" — ADB, the agent, models, and image processing all stay on the device host. It supports strongly typed Pydantic structured outputs and assertions, and plugs directly into frameworks like pytest.

4. MCP Integration Configuration

Using Claude Desktop as an example (claude_desktop_config.json):

{
  "mcpServers": {
    "artemis": {
      "command": "/path/to/artemis/.venv/bin/python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/artemis"
    }
  }
}

The docs also recommend mounting a behavior rules file (mcp_server/rules.md) for AI agents, covering proactive exploration before coding, Flash/Pro routing strategy, and latency compensation logic.

A Deeper Look

Flash vs. Pro: A Pragmatic Speed/Reliability Trade-off

ARTEMIS doesn't use a single mode for every task — it explicitly splits into two architecturally distinct execution paths:

Flash mode (reactive):
  Observe screen → think about next step → execute action → loop
  ↑ Single-model loop, no planning, no safety net, no final report
  ↑ Loop count is unbounded by default (history is compressed, not truncated)
  ↑ Good for short, deterministic tasks like "open Settings and flip a switch"

Pro mode (plan-and-verify):
  Planner drafts a Markdown task plan (with milestones and verification items)
      ↓
  Operator runs a "Safety Net" check before each action
      ↓ (checks against the live UI tree first, then pixel fallback)
  Hits a blocker → opens an "execution incident," Operator handles recovery autonomously
      ↓
  Checker (read-only) verifies plan checkpoints, supports a final review before exit

The core insight behind this design: mobile automation task complexity is bimodal. A large share of tasks are deterministic short tasks — "open a page and confirm a state" — where a simple reactive loop is fast and sufficient. A minority are complex workflows that need multi-step planning, fault tolerance, and long-running monitoring, where only a multi-agent plan-execute-verify structure can guarantee reliability. Using one architecture for both extremes wastes something either way.

Fault-Tolerant Design in Multi-Modal Target Localization

ARTEMIS's element localization mechanism reflects a "prefer precision, degrade gracefully" fault-tolerance philosophy:

Localization priority:
  1. Accessibility Hierarchy — most precise, the go-to for standard Android UI
  2. OCR text recognition — handles cases where the hierarchy tree lacks information
  3. Vision model recognition — fallback, handles custom Canvas/Compose/Flutter UIs

Standard Android widgets are usually exposed accurately by accessibility services, making that the fastest and most reliable localization method. But more and more apps render their UI with Compose, Flutter, or custom Canvas drawing, which is unfriendly to accessibility services — that's when OCR and vision models step in. This layered fallback strategy avoids the waste of "use the most expensive method for everything" while also avoiding the fragility of "just fail outright when standard widgets aren't recognized."

Shared History Compression: Memory Management for Long-Running Tasks

Flash and Pro share the same history compression mechanism, and this detail is worth noting:

Problem: long-running tasks (especially Pro's 100+ step workflows) accumulate
a large volume of screenshots and action logs
  ↑ Stuffing them directly into context blows past the window limit
  ↑ Simple truncation loses important early information

ARTEMIS's approach:
  Old screenshots → replaced with visual summaries (keep semantics, drop pixel detail)
  Completed steps → compressed into retrievable history chunks
  ↑ The agent can retrieve past context when it needs to look back,
    instead of always carrying the full history

This design lets Flash mode run stably long-term even with no loop cap, and is also the memory-management foundation that makes Pro's 100+ step workflows possible.

How It Compares to Similar Mobile Automation Approaches

Dimension	Appium (traditional scripts)	Pure vision-model-driven	ARTEMIS
Localization method	Fixed element ID/XPath	Pure visual recognition	Element index first + OCR + vision fallback
UI-change tolerance	❌ Brittle	✅ Fairly robust	✅ Fairly robust
Execution speed	Fast	Slow	Flash is fast / Pro is moderate
Natural-language driven	❌	Partial support	✅ Native support
AI IDE integration (MCP)	❌	Uncommon	✅ Native support across multiple IDEs
Long-running task plan/verify	Must build yourself	Uncommon	✅ Native in Pro mode
Open source	✅	Varies by project	✅ Apache-2.0

ARTEMIS's differentiation is combining "natural-language driven + multi-modal fault-tolerant localization + native AI IDE integration" into one tool, using its dual-mode design to avoid the dilemma of "either too slow with pure vision models, or too brittle with fixed scripts."

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/google/artemis
📄 License: Apache License 2.0
🐛 Issues: GitHub Issues

Related Resources

AndroidWorld — Google Research's mobile agent benchmark project, and ARTEMIS's evaluation benchmark
Model Context Protocol — The standard protocol underlying ARTEMIS's integration with Claude Code and other AI IDEs
mobile-use — The open-source project (from Minitap, Inc.) that ARTEMIS is built on

Summary

Key Takeaways

Natural-language-driven mobile automation: replaces brittle element-locator scripts with instructions, lowering test maintenance cost
Dual Flash/Pro mode design: choose a fast reactive loop or a plan-execute-verify workflow based on task complexity, balancing speed and reliability
Multi-modal fault-tolerant localization: accessibility hierarchy first, OCR and vision models as graceful fallbacks, adapting from standard widgets to custom Canvas UIs
Native MCP integration: gives Claude Code, Antigravity, Codex, and other AI IDEs the ability to directly operate real Android devices
99%+ on AndroidWorld: reliability validated on a standard mobile agent benchmark

Who This Is For

Mobile app testing teams: want to escape brittle element-locator scripts and reduce maintenance cost from UI changes
AI application developers: want AI coding assistants to be able to operate real Android devices for feature verification or bug reproduction
CI/CD pipeline maintainers: need to seamlessly fold mobile automation testing into existing test frameworks and continuous integration
Exploratory/stability testing teams: need long-running, continuously monitored mobile testing capability

One-Line Verdict

ARTEMIS doesn't pick a side between "fast but brittle" and "robust but slow" — it splits the tension apart directly with its Flash/Pro dual-mode design, which may be a pragmatic middle answer as mobile automation moves toward AI-driven approaches.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 187): TeamAI-CLI — Tencent's Team-Level AI Agent Middleware, Turning Individual AI Skills Into Shared Team Assets

WonderLab — Sun, 20 Sep 2026 02:28:57 +0000

Introduction

"An AI agent's capabilities shouldn't be trapped in one person's chat history — they should become a shared asset for the whole team."

This is the 187th article in the "One Open Source Project a Day" series. Today's project is TeamAI-CLI.

When everyone on a team is using Claude Code, Cursor, or Codex to write code, a hidden problem starts building up: everyone is independently "training" their own AI. One engineer spends half a day figuring out the right prompting approach to get Claude Code to correctly understand the project's architecture; another hits the exact same wall with no idea it's already been solved, and has to work through it from scratch. Skills, Rules, and CLAUDE.md configs scattered across the team never converge into anything shared or synchronized.

That's exactly the problem TeamAI-CLI addresses. It's not another AI coding assistant — the project's own documentation lists Claude Code, Codex, and Cursor as tools it's compatible with, not competitors. It's a meta-management layer that sits on top of the AI tools your team already uses, responsible for syncing skills, rules, and knowledge bases across members, and for turning individual experience gained while using AI into a shared team asset.

4.8k Stars, MIT License, built in TypeScript.

What You Will Learn

TeamAI-CLI's three-layer product architecture: Team Execution / Team Context / Team Improvement
The push → MR → review-and-merge → SessionStart-hook-sync distribution flow
The friction-signal-driven mechanism for automatically capturing experience
How the codebase knowledge graph is built (tree-sitter AST + regex heuristics)
Its relationship to Claude Code, Codex, and similar tools: not a replacement, but a "team layer"

Prerequisites

Some experience using AI coding tools like Claude Code, Cursor, or Codex
Familiarity with Git workflows (branches, Merge Requests)
Optional: basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

TeamAI's official positioning is "the shared foundation for how your team works, learns, and improves with AI." The core idea, in one sentence: turn individual AI capabilities into shared team capabilities — across agents, machines, and team members.

Team and Background

Organization: Tencent
License: MIT License
Primary language: TypeScript
Distribution: global npm install (teamai-cli)

Project Stats

⭐ GitHub Stars: 4,800+
🍴 Forks: 340+
👀 Watchers: 17
📄 License: MIT
🔧 Open Issues: 33, Open PRs: 14 (actively maintained)

What It Does

The Problem It Solves

A team without TeamAI:
  One engineer built a perfect set of project rules in Claude Code  ← lives only on their machine
  Another spent three days figuring out how to get the AI to understand the architecture  ← that lesson never spreads
  Someone's CLAUDE.md and team conventions went stale long ago  ← nobody keeps them in sync
  ↑ Everyone's AI capability stays siloed — the team's collective ability never compounds

TeamAI's approach:
  A unified team resource repository (Skills/Rules/Docs/Agents/Hooks/MCP)
  ↓ teamai push (share) → MR review → merge
  ↓ SessionStart hook auto-triggers teamai pull (sync)
  Every team member's local AI tools automatically get the latest team standards
  ↑ Individual experience → team asset — AI capability can be version-controlled,
    reviewed, and distributed just like code

Use Cases

A tech lead standardizing AI tool conventions across the team
- Write project-specific coding standards and architecture constraints as Skills/Rules once, sync them to everyone
New members onboarding quickly into an existing AI collaboration setup
- After joining the team repo, local AI tools automatically pull the latest skills and context — no starting from scratch
Team members sharing personal experience
- Use teamai contribute to share a hard-won debugging insight with the whole team in one command
Codebase knowledge accumulation for AI retrieval
- The knowledge graph feature lets AI automatically retrieve relevant code context and historical decisions before tackling a coding task
Visualizing team-wide AI usage
- Use dashboard and digest commands to understand team-wide AI usage trends and activity

Quick Start

Option 1: Conversational install (let the AI set it up)

Paste this directly to Claude Code or any agent that supports Skills:

Install the teamai skill: https://github.com/Tencent/teamai-cli/tree/main/skills/teamai ,
load the teamai skill, then set up TeamAI for my team from scratch.

Option 2: Manual CLI install

# Admin: create the team's shared repository
npm install -g teamai-cli
teamai init https://github.com/yourorg/yourrepo

# Team member: join an existing team
cd /path/to/my-project
teamai init https://github.com/yourorg/yourrepo

# Or install at the user scope (applies across all projects)
teamai init https://github.com/yourorg/yourrepo --scope user

Core Features

1. A Three-Layer Product Architecture

Layer	Positioning	Core Capabilities
Team Execution	"Make every agent work the team's way"	init/pull/push sync for Skills/Rules/Agents/Hooks/MCP/Env
Team Context (beta)	"Make every agent understand the team"	Knowledge recall, learnings, codebase knowledge graph, team wiki
Team Improvement (beta)	"Make every execution improve the team"	Friction-signal-driven experience sharing, session logs, weekly digest, usage dashboard

2. A Git-Native Distribution Flow

teamai push → creates a branch + MR → reviewer approves and merges
                                              ↓
        SessionStart hook auto-triggers teamai pull → synced to local AI tools

Supports GitHub, GitLab, GitCode, CNB, TGit, and private Git servers — essentially folding AI collaboration standards into the normal code review process.

3. Key Commands

Command	Function
`teamai init`	Initialize: OAuth login, link repo, register member, inject hooks
`teamai pull` / `push`	Pull/push team resources
`teamai status`	Show the diff between local state and the team repo
`teamai contribute`	Share session experience to the team repo
`teamai recall <query>`	Search the team knowledge base (BM25 + graph-augmented)
`teamai codebase --extract/--deep-enrich/--reconcile`	Codebase knowledge graph operations
`teamai digest` / `dashboard`	Team usage stats and visualization
`teamai doctor`	Diagnose configuration issues

4. A Broad Agent Compatibility Matrix

TeamAI-CLI explicitly claims compatibility with over a dozen AI coding tools, including Claude Code, Codex, Cursor, GitHub Copilot CLI, CodeBuddy, OpenCode, and Kiro. The documentation includes a detailed compatibility table showing which capabilities (skills/rules/docs/env/hooks/mcp) each tool supports.

A Deeper Look

Friction-Signal-Driven Experience Capture

The most interesting design in TeamAI-CLI's Team Improvement layer is its trigger logic: instead of requiring users to proactively log their experience, it monitors friction signals in a session to detect "something worth recording just happened."

Examples of friction signals:
  "The user interrupted the AI twice"
  "The AI retried the same failing tool call 8 times in a row"
  ↓
  The system identifies unusual friction
  ↓
  It prompts the user: "This debugging session looks like it has something
  worth sharing — want to contribute it to the team?"

This is smarter than "asking employees to write weekly summaries of lessons learned" — most valuable debugging insights happen in the moment while solving a problem, and details fade fast in hindsight. Using interaction friction as the trigger captures experience while it's still fresh.

The Codebase Knowledge Graph: A Dual-Track Extraction Strategy

The codebase knowledge graph in the Team Context layer uses a dual-track strategy:

AST Track (precise, but costly)
  Uses a tree-sitter WASM parser to parse code
  ↓ precisely extracts functions, classes, and dependency relationships
  ↓ suited to deep parsing of core modules

Heuristic Track (fast, but coarse-grained)
  Scans code with regex heuristics
  ↓ quickly covers a broad range of files
  ↓ suited to fast indexing of peripheral code

Both tracks' results → written to the teamwiki/ directory → power the recall
command's BM25-based retrieval

This dual-track design reflects a practical trade-off: running tree-sitter-level precise AST parsing across an entire large codebase is expensive, while pure regex matching loses semantic relationships. Combining both — precise parsing for core modules, fast coverage for the periphery — is a pragmatic middle ground.

How It Relates to Claude Code and Codex

This is the point most likely to cause confusion about TeamAI-CLI, so it's worth clarifying directly:

Wrong framing: TeamAI-CLI competes with Claude Code, as another AI coding CLI
Correct framing: TeamAI-CLI sits on top of Claude Code/Codex/Cursor,
                  acting as a team-level distribution and knowledge management layer

Analogy:
  Claude Code / Codex / Cursor  ≈ the IDE on each person's machine
  TeamAI-CLI                    ≈ the team's Git repo + code review process
                                   (but for Skills/Rules/knowledge, not code)

It works by injecting hooks (like a SessionStart hook) and shared config files (CLAUDE.md, Skills directories, etc.) into already-installed AI tools. It doesn't provide reasoning capability itself — it orchestrates "who should use which rules, and when they get synced."

How It Compares to Similar Team Collaboration Tools

Dimension	Manually Maintained CLAUDE.md	Cursor Rules (local)	TeamAI-CLI
Cross-agent compatibility	Depends on each tool's own implementation	Cursor only	✅ A dozen+ tools
Team sync mechanism	Manual copy-paste / word of mouth	❌ None	✅ Git push/pull + MR review
Automatic experience capture	❌	❌	✅ Friction-signal-driven
Codebase knowledge graph	❌	❌	✅ AST + heuristic dual-track
Usage visualization	❌	❌	✅ dashboard/digest
Version control and review	Manual	Local files, no review	✅ Standard Git flow

TeamAI-CLI's differentiation is treating "AI collaboration conventions" as a first-class citizen of standard software engineering process (version control, code review) — instead of leaving everyone to maintain their own local config files.

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/Tencent/teamai-cli
📦 npm package: teamai-cli
📄 License: MIT License
🐛 Issues: GitHub Issues

Related Resources

tree-sitter — The parser framework behind TeamAI-CLI's codebase knowledge graph AST Track
Model Context Protocol — The resource distribution standard involved in TeamAI-CLI's compatibility matrix
Claude Code — One of the main AI coding agents TeamAI-CLI integrates with

Summary

Key Takeaways

Not an AI coding assistant, but a team-level middleware layer: sits on top of Claude Code/Codex/Cursor, focused on syncing standards rather than generating code
A three-layer architecture that builds progressively: Team Execution handles distribution, Team Context handles understanding, Team Improvement handles evolution
A Git-native distribution flow: push → MR review → merge → hook-triggered sync — AI collaboration conventions follow standard software engineering process
Friction-signal-driven experience capture: doesn't rely on manual summarization; interaction anomalies automatically trigger experience capture
Compatible with a dozen+ agents: not locked to a single AI tool ecosystem, covering mainstream choices like Claude Code, Codex, and Cursor

Who This Is For

Engineering team leads: want to standardize AI collaboration conventions across the team instead of letting everyone figure it out independently
Mid-to-large engineering teams: members use different AI tools and need a neutral layer to keep standards in sync
Teams that value knowledge management: want to systematically capture hard-won lessons and codebase knowledge instead of losing them to staff turnover
Teams already using Claude Code/Cursor and similar tools: don't want to switch tools, just want the ones they already use to be smarter and more consistent

One-Line Verdict

TeamAI-CLI isn't trying to solve "how AI writes code" — it's trying to solve "how one person's AI experience becomes everyone's experience," which may be a more interesting direction than simply stacking on stronger models.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

pen Source Project #186: security-audit — Cloudflare's Skill That Turns Your Coding Agent into a Six-Phase Security Auditor

WonderLab — Sat, 19 Sep 2026 02:55:33 +0000

Introduction

"A coding-agent skill for multi-phase security audits with independently verified, machine-readable findings."

This is article #186 in the "One Open Source Project a Day" series. Today's project is security-audit — Cloudflare's coding-agent skill, 13,825 Stars, MIT license.

security-audit addresses a critical question: how do you make AI-generated security audit results trustworthy enough to hand to a security team? The answer isn't "a smarter model" — it's a structured process: six audit phases, adversarial validation, and machine-readable finding records. It turns "the model says there's a problem here" into "here's a confirmed vulnerability with source evidence, a reproducible path, and a priority ranking."

What You'll Learn

The six-phase audit workflow, from reconnaissance to target-neutral reporting
The adversarial validation principle (the checker is never the finder)
The difference between three verdicts: confirmed / needs_validation / rejected
How the coverage ledger makes multi-run audits additive
The sandbox requirement: why "can't execute target code" means needs_validation

Prerequisites

Experience with Claude Code or similar coding agents and the skill mechanism
Basic understanding of security audit concepts (attack surface, trust boundaries, vulnerability confirmation)
Node.js fundamentals

Project Background

Overview

This skill is the single-repo starting point of Cloudflare's vulnerability discovery harness. Cloudflare's official blog Build your own vulnerability harness describes how that system evolved into a multi-stage, fleet-wide vulnerability discovery platform — and security-audit is the single-repo version it evolved from.

It's not a "generate a security report" prompt template — it's an orchestration system: isolated sub-agents run reconnaissance, hunting, validation, and verification, and every step produces structured records with independent validators.

Author / Team

Organization: Cloudflare
Primary language: JavaScript (zero-dependency validators, written for Node.js)
License: MIT
Created: 2026-06-18

Project Stats

⭐ GitHub Stars: 13,825+
🍴 Forks: 740+
📄 License: MIT
📅 Created: 2026-06-18

Quick Start

Installation

# Install with the Skills CLI
npx skills add https://github.com/cloudflare/security-audit-skill \
  --skill security-audit

# User-level installation
npx skills add https://github.com/cloudflare/security-audit-skill \
  --skill security-audit \
  --global

Usage

Start your coding agent pointed at the codebase you want to audit, then say:

security audit this codebase

Or:

find security vulnerabilities in ./src

do a security review, output to ~/audits/my-project

The skill activates automatically when the request matches its trigger (security audit / find vulnerabilities / pen-test, etc.).

Two Modes

Mode	Trigger	Behavior
guidance	Security questions, focused reviews, methodology	Use only relevant parts; don't run the full workflow or write files
full audit	Explicit audit/pen-test request, end-to-end review	Run all six phases and write report files

Core: The Six-Phase Audit Workflow

Phase 1: Reconnaissance

Launch multiple research agents in parallel, each returning structured source facts with file:line references:

Agent 1a: product type, tech stack, build commands, subsystem boundaries
Agent 1b: principals, authority, trust boundaries, control locations
Agent 1c: entry surfaces, copies, and sinks

Outputs architecture.md and coverage-ledger.json. Reconnaissance is read-only — no external services are contacted.

Phase 2: Coverage-Led Hunting

Assigns ledger "coverage units" to isolated general agents. Each hunter reads only its assigned source blocks, writes only to its own scratch/, and returns one structured result.

Key: coverage critics find gaps — which units were missed, which assignments overlap.

Phase 3: Independent Candidate Validation

Every unique candidate (after deduplication) goes to a fresh verifier that did not hunt it, tasked with trying to refute it.

The verifier's prompt says explicitly: "You did not write this candidate. Try to refute it from repository source and bounded local evidence."

Phase 4: Structured Output

Writes three verdict records to findings.json and validates them with validate-findings.cjs.

Phase 5: Independent Record Verification

Fresh agents verify final source claims. Material replacements receive another independent verifier.

Phase 6: Target-Neutral Reporting

Derives REPORT.md, FINDINGS-DETAIL.md, and NEEDS-VALIDATION.md from the verified records and coverage ledger.

Three Verdicts: Clear Semantics

This is the most instructive design point — verdicts aren't "high/medium/low risk"; they're "evidence completeness":

Verdict	Meaning
`confirmed`	Complete source trace with a bounded, reproducible observation
`needs_validation`	An exact unresolved fact, but no severity assigned
`rejected`	A disproved candidate (records why it was rejected)

The key principle: "a source-grounded suspicion" ≠ "a confirmed vulnerability." When a lead can't be validated because of sandbox constraints, it stays needs_validation — never hastily marked confirmed, never silently dropped.

Design Principles

1. Adversarial Validation

The agent that checks a finding is never the agent that found it. This prevents model self-confirmation — when a model validates its own finding, it tends to confirm it.

2. Severity Requires Impact

Severity = likelihood × impact, not deviation from a checklist. A problem that matches a checklist but has no actual impact is not a vulnerability.

3. Defense-in-Depth Gaps Are Not Vulnerabilities

If Layer A already prevents the attack, the absence of Layer B is a hardening note, not a vulnerability.

4. Multiple Runs Improve Coverage

In Cloudflare's test runs, a single run found roughly half the vulnerabilities that repeated runs found in total. So the skill is designed for additive multi-run coverage — each run uses prior ledgers and findings to target gaps.

The Coverage Ledger: Making Audits Additive

coverage-ledger.json is the skill's core data asset.

A normal security audit is one-shot — run once, produce a report, and running again means starting from zero. security-audit's ledger makes audits incremental:

First audit run:
  → generates coverage-ledger.json (records which units are covered, confirmed, or pending)
  → finds N vulnerabilities

Second audit run (same repo):
  → reads the prior ledger and findings
  → only re-hunts "uncovered gaps" and "changed source"
  → carries forward prior findings still backed by current source
  → does NOT treat stale or unresolved prior work as covered

Each unit has a state: planned → in_progress → completed / deferred. If a unit can't be assigned because of agent-count limits, it's explicitly marked deferred with a reason — never silently dropped.

Machine-Readable Finding Records

findings.json follows the schema in report-schema.json, paired with zero-dependency validators:

File	Purpose
`report-schema.json`	JSON schema for all three verdicts
`validate-findings.cjs`	Zero-dependency validator (used in Phases 4/5)
`validate-coverage-ledger.cjs`	Coverage ledger validator (used in Phases 1–5)

The parent runs validate-coverage-ledger.cjs after creating the ledger and after each update, and validate-findings.cjs in Phase 4 and after each Phase 5 replacement.

This "machine-readable + independently validated" design lets audit output feed downstream tooling, rather than being a PDF only humans can read.

Sandbox Requirements: An Honest Boundary

security-audit explicitly requires: executing target-controlled code must happen inside an OS-enforced sandbox.

The sandbox must:

Disable external networking
Use a sanitized allowlisted environment
Enforce resource limits
Allow writes only to assigned scratch paths

If these controls are unavailable, the workflow does not execute target code and keeps the lead as needs_validation.

This is an honest engineering decision: better to leave something unconfirmed than to run potentially malicious code without a safe boundary. In AI security tooling, this explicit stance on the "validation boundary" is more trustworthy than "I can auto-run the PoC."

Resources

🌟 GitHub: cloudflare/security-audit-skill
📝 Background blog: Build your own vulnerability harness
🔧 Skills CLI: skills.sh
✉️ Contact: security-ai-research@cloudflare.com

Summary

security-audit represents a specific judgment: the bottleneck in AI security auditing isn't "can the model find vulnerabilities" — it's "can the findings be trusted."

Three things worth noting:

Adversarial validation is the core of credibility. The problem with many AI audit tools is that discovery and validation are done by the same model — the model "finds" a vulnerability, then validates its own finding, naturally biased toward confirmation. security-audit cuts this bias at the process level with the "checker ≠ finder" structural constraint.

Verdict semantics = evidence completeness, not risk level. confirmed / needs_validation / rejected describe "where the evidence chain breaks," not "how severe this is." Severity (likelihood × impact) is a field inside confirmed. This separation makes audit results machine-processable and trusted by security teams.

Additive coverage solves the "audits are one-shot" problem. Traditional audits go stale the moment they're done. security-audit's coverage ledger makes audits incremental: each run builds on prior coverage, only filling gaps and revalidating changes. This is closer to "continuous security" than "periodic audit."

If you're doing security work with a coding agent, or want to understand how to make AI-generated security conclusions trustworthy, security-audit is the most complete open-source reference available.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

One Open Source Project a Day (No. 185): Coder — Self-Hosted Cloud Dev Environments, Now Also a Safe Workshop for AI Agents

WonderLab — Fri, 18 Sep 2026 02:17:44 +0000

Introduction

"Give every developer a consistent, isolated, disposable dev machine — now give every AI agent the same kind of workshop."

This is the 185th article in the "One Open Source Project a Day" series. Today's project is Coder.

"It works on my machine" is a classic problem every team has hit — a new hire spends three days configuring an environment, a dependency won't install after an OS upgrade, or the local dev box simply doesn't have enough resources to run a large project. Cloud Development Environments (CDEs) answer this problem: move the dev environment to the cloud, define it declaratively with Terraform, and make it consistent, reproducible, and disposable on demand.

But Coder has picked up a second layer of meaning in recent years. As AI coding agents (Claude Code, Codex, Cursor, and others) start executing tasks autonomously, a new question shows up: where does the agent actually run? Where does its API key live? Who can see what it did?

Coder's answer: run agents on the same isolated, auditable infrastructure as human developers. The agent's reasoning loop runs on the control plane, and no LLM credentials ever sit inside the workspace — this is a security boundary designed for AI agents from the start, not a bolt-on fix.

14.9k Stars, AGPL-3.0, built in Go.

What You Will Learn

How Coder uses Terraform to define cloud development environments
The WireGuard-tunnel network architecture connecting workspaces
Four ways to run AI agents: Coder Agents, Agent Relay, Agents in the IDE, and Agents in workspace templates
How the AI Gateway centralizes model governance, cost tracking, and auditing
How Coder's positioning differs from traditional CDEs like GitHub Codespaces and Gitpod

Prerequisites

Basic understanding of Terraform (Infrastructure as Code)
Familiarity with basic Docker/Kubernetes operations
Optional: familiarity with AI agent tool-calling and MCP concepts

Project Background

What It Is

Coder's official positioning is "Self-Hosted Cloud Development Environments and AI Agents." The GitHub description is even more direct: "Secure environments for developers and their agents."

That line captures Coder's current dual identity: a traditional cloud development environment platform, and a secure runtime foundation for AI agents.

Team and Background

Repository: coder/coder
Website: coder.com
License: AGPL-3.0 (enterprise features are covered by a separate license, see the repo's LICENSE.enterprise)
Primary languages: Go (backend) + TypeScript (frontend, in the site directory)

Project Stats

⭐ GitHub Stars: 14,900+
🍴 Forks: 1,500+
👀 Watchers: 82
📄 License: AGPL-3.0
🌐 Website: coder.com

What It Does

The Problem It Solves

Traditional dev environment setup:
  New hire → install system deps → configure IDE → set up DB → get running
  ↑ Takes days, and "works on my machine" issues are common
  ↑ Local machine may lack the resources for large projects

Cloud development environments (Coder's approach):
  Terraform template defines the environment → developer clicks a button → consistent env in seconds
  ↑ Onboarding time drops from days to seconds
  ↑ Idle resources shut down automatically, saving cost
  ↑ Environments are fully consistent — no more "my machine" problem

The new problem in the AI agent era:
  You want Claude Code / Codex to run tasks autonomously
  → Where does it actually run? Locally? Inside the workspace?
  → Where does the API key live? Is exposing it to the workspace safe?
  → Who's tracking how many tokens the agent spent and what it did?

Coder's answer:
  The agent's reasoning loop runs on the control plane (on your own infrastructure)
  → No LLM credentials ever live inside the workspace
  → Every action is tied to a specific user identity — auditable by design
  → The AI Gateway centralizes authentication, cost, and policy

Use Cases

Standardized development environments for teams/enterprises
- Replace "everyone configures their own machine" — new hires get a standardized workspace in seconds
Cloud-based development for large projects
- Offload compute-heavy work to the cloud when local hardware isn't enough; the local machine is just a lightweight client
Hosting AI agent background tasks
- Delegate long-running tasks to Claude Code, Codex, and other coding agents inside isolated workspaces without tying up the developer's local machine
Enterprise-grade AI model governance
- Scenarios that need to track team-wide token usage, which agents called which tools, and centrally manage API keys
Environments with strict compliance and audit requirements
- Industries like finance and healthcare that need complete operational audit logs for their dev environments

Quick Start

# Quick local trial
curl -L https://coder.com/install.sh | sh
coder server

# Production deployment (requires PostgreSQL 13+)
coder server --postgres-url <postgres-connection-string> --access-url <public-url>

A Helm Chart is also available for Kubernetes deployment, suited to team-scale production use.

Core Features

1. Terraform-Defined Workspaces

Each workspace's underlying infrastructure is declared through a Terraform template, with support for:

Backend	Description
AWS	EC2 instances and other cloud resources
Kubernetes	Pod-level workspaces
Docker	Containerized workspaces, good for local/small-scale deployments
Other cloud platforms	Extendable through the Terraform Provider ecosystem

The Template Builder offers a guided UI for common configurations, so you don't need to hand-write Terraform for everything.

2. WireGuard Secure Tunnels

Workspaces connect over WireGuard tunnels, keeping network traffic secure while supporting flexible topologies — direct SSH, IDE remote connections, or the web terminal.

3. Automatic Shutdown of Idle Resources

Workspaces automatically shut down their underlying compute resources when unused, starting them back up only on demand — a key cost control mechanism that avoids the classic "forgot to turn off the cloud VM" problem.

4. Four Ways to Run AI Agents

Method	Best For
Coder Agents	A built-in native AI coding agent whose reasoning loop runs on the control plane; the workspace can be fully network-isolated, ideal for long-running background tasks
Agent Relay	Connects a cloud-hosted agent service (like Cursor Cloud Agents) to a self-hosted workspace — reasoning happens in the cloud, the workspace executes tool calls (early preview)
Agents in the IDE	Integrates with IDE-native agents like Cursor, Devin Desktop, and Zed, working alongside a developer's existing workflow
Agents in workspace templates	Template admins install terminal-based agents like Claude Code or Codex directly into workspace templates via Registry modules

An example of an agent baked into a template:

module "claude-code" {
  source   = "registry.coder.com/coder/claude-code/coder"
  version  = "~> 5.2"
  agent_id = coder_agent.main.id
}

5. AI Gateway (Enterprise)

Centralized management of all AI model access:

Unified authentication: no need to configure API keys separately in every workspace
Audit trail: logs all prompts and tool calls
Policy enforcement: governs access to upstream LLM providers
Agent Firewall: process-level network and command policies that constrain what an agent can access and execute inside a workspace

6. Editor and Ecosystem Integrations

VS Code extension, JetBrains Toolbox plugin
Dev Containers support (@devcontainers/cli + Docker)
Envbuilder (an alternative for environments without Docker access)
Kubernetes log streaming, GitHub Actions integration

A Deeper Look

Why Running the Agent Loop on the Control Plane Matters

This is the most interesting architectural choice in Coder's AI agent design. Most AI coding tools follow this pattern:

Common architecture (agent runs inside the workspace/locally):
  Developer machine / workspace
  ├── AI agent process
  ├── LLM API key (in an env var/config file)
  └── Tool calls execute directly, locally

  Risk: if the workspace is compromised, the API key is exposed directly
  Risk: there's no unified audit point for everything the agent does

Coder Agents takes a different approach:

Coder's architecture (agent reasoning runs on the control plane):
  Control plane (your infrastructure, not the workspace)
  ├── Agent reasoning loop
  ├── LLM API key (lives here, and only here)
  └── Decisions → dispatched to the workspace for execution

  Workspace
  ├── Only receives specific execution instructions
  ├── Never stores any credentials
  └── Execution results → reported back to the control plane

  Benefits:
  - A compromised workspace leaks no API key (it's simply not there)
  - Every tool call is tied to a user identity — auditable by design
  - Workspaces can be fully network-isolated and the agent still works

The design essentially separates "brain" from "hands and feet" — reasoning happens in a controlled central environment, execution is distributed to isolated workspaces. For enterprise scenarios that need compliance audits, this architecture gives you a single, controllable audit point.

The Trade-offs Across Four Agent Modes

Coder doesn't push a single way to integrate AI agents — it offers four choices, each mapping to a different trust model and use case:

Ranked by trust level, highest to lowest:

1. Coder Agents (most controlled)
   Reasoning happens entirely on your infrastructure; credentials never leave the control plane
   Best for: enterprises with strict compliance needs requiring full audit trails

2. Agents in workspace templates (controlled + flexible)
   The agent itself runs inside the workspace, but the workspace is standardized and auditable
   Best for: teams that need to use specific tools like Claude Code or Codex

3. Agents in the IDE (developer autonomy)
   Follows the developer's own local IDE configuration; Coder just provides the underlying workspace
   Best for: letting developers use familiar tools while Coder handles the infrastructure

4. Agent Relay (trusting a third-party cloud service)
   Orchestration/reasoning happens on a third-party cloud, the workspace just executes
   Best for: teams that want cloud agent capabilities like Cursor Cloud, but need self-hosted execution

The practical implication of this layered design: different teams have different trust boundaries for "what can the AI agent see, what can it do." Coder covers the entire spectrum — from "fully closed internal loop" to "bring in external cloud intelligence" — on the same underlying infrastructure.

How It Compares to Similar Cloud Development Environments

Dimension	GitHub Codespaces	Gitpod	Coder
Deployment	GitHub-hosted only	Mostly cloud-hosted	Fully self-hosted
Infrastructure definition	Limited customization	`.gitpod.yml`	Terraform (full IaC)
Data sovereignty	❌ Lives on GitHub's cloud	Partial	✅ Fully self-controlled
Native AI agent support	Limited	Limited	✅ Four modes
Model governance (AI Gateway)	❌	❌	✅ Enterprise
Backend flexibility	Fixed	Fixed	Choose AWS/K8s/Docker
Open source	❌	Partially open	✅ AGPL-3.0

Coder's core differentiation: data sovereignty from full self-hosting + Terraform-level infrastructure flexibility + a security boundary purpose-built for AI agents. The trade-off is you have to operate the system yourself rather than getting "it just works" like Codespaces.

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/coder/coder
🌐 Website: https://coder.com
📚 Docs: https://coder.com/docs
💬 Discord: Coder's community Discord
🐛 Issues: GitHub Issues

Related Resources

Terraform — The foundation of Coder's template system, declarative infrastructure definition
Devcontainers — The containerized dev environment standard Coder supports
Model Context Protocol — The tool-calling standard relevant to Coder's agent ecosystem

Summary

Key Takeaways

Terraform defines everything: workspace infrastructure is declared as IaC, freely choosing AWS/Kubernetes/Docker backends, fully reproducible
WireGuard secure tunnels + automatic idle shutdown: network security and cost control together
Four AI agent integration modes: from the fully controlled Coder Agents to the flexible Agents in the IDE, covering different trust boundaries
Agent reasoning runs on the control plane: credentials never enter the workspace, auditable by design — a security architecture built for the AI agent era, not a feature bolted onto a traditional CDE
The AI Gateway provides centralized governance: unified authentication, cost tracking, and policy enforcement — necessary infrastructure for enterprise-grade AI usage

Who This Is For

Teams that need a self-hosted cloud dev environment: don't want code and data on a third-party cloud, but still want Codespaces-level experience
Organizations running AI coding agents at scale: need to centrally manage API keys, track cost, and audit agent actions
Enterprises with strict compliance and audit requirements: finance, healthcare, and other industries that need complete operational logs
DevOps/platform teams: want to manage dev environment infrastructure uniformly with Terraform instead of everyone configuring their own

One-Line Verdict

Coder originally solved the old problem of "cloud development environments," but the security boundary it built for AI agents — reasoning on the control plane, credentials never in the workspace — may turn out to be its real value in this era.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

DeepSeek Harness Series (10): Building a Complete dsh Plugin — From Requirements to Production

WonderLab — Fri, 18 Sep 2026 02:01:13 +0000

From 'Knowing' to 'Doing'

The previous nine articles covered a lot of ground: the Cordis plugin system, tool registration, the Agent loop, Session memory, System Prompt assembly, capability seams, multi-Agent collaboration, observability...

Knowing what each piece does, and actually assembling them into something that runs, are two different things.

This article does the second one.

We'll build a complete plugin from scratch — from the requirements description to code you can drop into a Bundle. The knowledge from the previous nine articles won't be recited in a list — it will show up naturally in the code. Where something needs explanation, there's a note with a chapter reference.

What We're Building

Plugin name: workspace-context

Feature requirements:

Inject the current working directory into the system prompt (so the model 'knows where it is')
Provide a get_workspace_info tool (so the model can actively query workspace files)
Pause before dangerous tool calls (names containing delete or rm) and wait for user confirmation
Measure execution time for every tool call (observability)
When the plugin is unloaded, all registrations clean up automatically

These five features map directly to the five main concerns of plugin development: prompt + tools + permissions + observability + lifecycle.

The Plugin Skeleton

A dsh plugin is a TypeScript module that follows the Cordis plugin protocol:

// packages/workspace-context/src/index.ts

// Plugin name: unique identifier in the Cordis dependency tree
export const name = 'workspace-context'

// Declare service dependencies.
// Cordis ensures these services are ready before calling apply.
// If a service is unavailable, the plugin suspends (no crash — just waits).
export const inject = ['tools', 'systemPrompt']

// Plugin entry point.
// ctx is the private Cordis context for this plugin.
// Everything registered through ctx (tools, prompt sections, event listeners)
// is bound to this context's lifecycle — automatically disposed when the plugin unloads.
export function apply(ctx: Context): void {
  // All registration goes here
}

These three declarations (name, inject, apply) are the minimum structure for a dsh plugin.

inject isn't just an import list — it's a contract. It tells the Cordis runtime: 'I depend on these services. If they're not ready yet, don't call me.' This is the foundation for hot-swapping (covered in article 02 on the Cordis plugin system).

Step 1: Inject a System Prompt Section

The cleanest way to make the model aware of the current working directory is through a System Prompt Section — not by repeating it in every user message (article 06: System Prompt Assembly).

// Inject workspace information into the system prompt
ctx.systemPrompt.section({
  name: 'workspace-context:cwd',
  // Place this section after Agent instructions, before user-facing prompts
  order: ctx.systemPrompt.getSectionOrder('context:workspace'),
  // Dynamic text: re-evaluated every time the system prompt is assembled.
  // If the working directory changes mid-session, the next request gets the fresh value.
  text: (context) => {
    const cwd = context.agent?.session?.header?.cwd
    if (!cwd) return ''  // No cwd available — inject nothing
    return [
      '## Workspace',
      `Current working directory: ${cwd}`,
      'All relative file paths are resolved from this directory.',
    ].join('\n')
  },
})

Notice that text is a function, not a string. dsh calls it each time it assembles the system prompt — so the injected information is always current, never a stale snapshot.

One more detail: we don't need to save the return value of section() (which is a disposer). Because we used ctx.systemPrompt.section(), Cordis tracks that this Section belongs to the current plugin's context and will clean it up automatically when the plugin unloads.

Step 2: Register the `get_workspace_info` Tool

A tool lets the model actively 'ask' rather than just 'be told' (article 03: Tool System).

import { defineTool } from '@deepseek-ai/dsh-tools'
import { readdir, stat } from 'node:fs/promises'
import { join } from 'node:path'

ctx.tools.register(defineTool({
  name: 'get_workspace_info',
  description: 'Get information about the current workspace: directory listing and file sizes.',

  parameters: {
    path: {
      type: 'string',
      description: 'Relative path within the workspace. Defaults to workspace root.',
    },
  },

  output: {
    schema: {
      // Strict mode: additionalProperties: false prevents unexpected fields from the model
      type: 'object',
      additionalProperties: false,
      properties: {
        path: { type: 'string' },
        entries: {
          type: 'array',
          items: {
            type: 'object',
            additionalProperties: false,
            properties: {
              name: { type: 'string' },
              type: { type: 'string', enum: ['file', 'directory'] },
              size: { type: 'number' },
            },
          },
        },
      },
    },
    // render converts the structured return value into model-readable text.
    // Separating 'structured data' from 'what the model sees' is a core dsh design principle.
    render: (_args, value) => {
      const v = value as {
        path: string
        entries: Array<{ name: string; type: string; size: number }>
      }
      const lines = [`Directory: ${v.path}`, '']
      for (const entry of v.entries) {
        const sizeStr = entry.type === 'file' ? ` (${entry.size} bytes)` : '/'
        lines.push(`  ${entry.name}${sizeStr}`)
      }
      return [{ type: 'text', text: lines.join('\n') }]
    },
  },

  // Read-only operation. Doesn't modify any state.
  // Safe to run in parallel with other tools.
  isConcurrencySafe: () => true,

  async execute(args, exec) {
    // Pull cwd from the session header; fall back to process.cwd() if absent
    const cwd = exec.agent?.session?.header?.cwd ?? process.cwd()
    const targetPath = args.path ? join(cwd, args.path) : cwd

    const entries = await readdir(targetPath, { withFileTypes: true })
    const result = await Promise.all(
      entries.map(async (entry) => {
        const fullPath = join(targetPath, entry.name)
        // Only call stat on files (to get size) — directories don't need it
        const stats = entry.isFile() ? await stat(fullPath) : null
        return {
          name: entry.name,
          type: entry.isDirectory() ? 'directory' : 'file',
          size: stats?.size ?? 0,
        }
      })
    )

    return {
      path: targetPath,
      // Directories first, then files; alphabetical within each group
      entries: result.sort((a, b) => {
        if (a.type !== b.type) return a.type === 'directory' ? -1 : 1
        return a.name.localeCompare(b.name)
      }),
    }
  },
}))

isConcurrencySafe: () => true is a performance signal. When dsh sees multiple tool call requests in the same Agent loop turn, it can run concurrency-safe tools in parallel instead of serializing them. For read-only tools, this should almost always be true.

Step 3: Permission Interception

Dangerous operations — deletions, overwrites — should give a human a chance to intervene before they run. dsh handles this via the tools/pre-execute event (article 03):

// Intercept tool calls whose names contain 'delete' or 'rm'.
// Returning { kind: 'ask' } pauses execution and waits for user confirmation.
// Returning { kind: 'deny' } rejects the call immediately — the tool never runs.
ctx.on('tools/pre-execute', async (exec, next) => {
  const isDangerous = exec.name.includes('delete') || exec.name.includes('rm')

  // Not dangerous: let it through, continue the execution chain
  if (!isDangerous) return next()

  // Dangerous: require confirmation.
  // If no approval service is configured in the Bundle,
  // 'ask' automatically degrades to 'deny'.
  return {
    kind: 'ask',
    reason: `About to run: ${exec.name}(${exec.arguments})`,
  }
})

This is the middleware pattern: next() means 'continue the chain'. Returning something else without calling next() short-circuits the chain — the tool is never executed.

There's also a design intent here: instead of a whitelist of safe tool names, the check is based on naming conventions. Real projects might need more precise rules, but the shape is the same.

Step 4: Execution Time Metering

In production, you need to know which tools are slowest and which calls take suspiciously long (article 09: Observability):

// Wrap every tool execution to measure elapsed time.
// tools/execute is a middleware hook — every tool call passes through it.
ctx.on('tools/execute', async (exec, next) => {
  const start = performance.now()

  // Call next() to run the actual tool implementation
  const result = await next()

  const elapsed = Math.round(performance.now() - start)

  // result.isError is the standardized error flag.
  // Here we log to console; in a real project, route to ctx.sessionTelemetry.
  console.log(`[${exec.name}] ${result.isError ? 'ERROR' : 'OK'} ${elapsed}ms`)

  // Must return result — otherwise the tool's return value is lost
  return result
})

A common mistake to watch out for: forgetting return result. The tools/execute hook is a 'wrapping' hook — you can do work before and after, and optionally transform the result, but you must pass it back.

Step 5: Session Observation

Log a summary at the end of each Turn for later analysis:

// Listen to session/event and print a summary when each Turn ends
ctx.on('session/event', (session, event) => {
  if (event.type !== 'turn/end') return

  // reason.kind is why the Turn ended:
  // 'complete'    — normal completion (model decided the task is done)
  // 'error'       — something went wrong
  // 'interrupted' — user interrupted
  // 'max-turns'   — hit the max turn count limit
  const reason = event.data.reason.kind
  const turnNum = event.data.turn

  console.log(`[Turn ${turnNum}] ${reason}`)
})

This is the minimal version. A complete production implementation would also emit a telemetry record via ctx.sessionTelemetry.emit() and print a token consumption summary using ctx.tokenMeter.measure(session).

The Complete Plugin: Putting It All Together

All five parts combined into one file:

// packages/workspace-context/src/index.ts
// workspace-context plugin: workspace awareness + tool + permission interception + observability

import { defineTool } from '@deepseek-ai/dsh-tools'
import { readdir, stat } from 'node:fs/promises'
import { join } from 'node:path'

// ── Plugin metadata ────────────────────────────────────────────
export const name = 'workspace-context'

// Declare service dependencies
// tools:        tool registry
// systemPrompt: system prompt section manager
export const inject = ['tools', 'systemPrompt']

// ── Plugin entry point ─────────────────────────────────────────
export function apply(ctx: Context): void {

  // ── 1. Inject a System Prompt Section ─────────────────────────
  ctx.systemPrompt.section({
    name: 'workspace-context:cwd',
    order: ctx.systemPrompt.getSectionOrder('context:workspace'),
    text: (context) => {
      const cwd = context.agent?.session?.header?.cwd
      if (!cwd) return ''
      return [
        '## Workspace',
        `Current working directory: ${cwd}`,
        'All relative file paths are resolved from this directory.',
      ].join('\n')
    },
  })

  // ── 2. Register the get_workspace_info tool ────────────────────
  ctx.tools.register(defineTool({
    name: 'get_workspace_info',
    description: 'Get information about the current workspace: directory listing and file sizes.',
    parameters: {
      path: {
        type: 'string',
        description: 'Relative path within the workspace. Defaults to workspace root.',
      },
    },
    output: {
      schema: {
        type: 'object',
        additionalProperties: false,
        properties: {
          path: { type: 'string' },
          entries: {
            type: 'array',
            items: {
              type: 'object',
              additionalProperties: false,
              properties: {
                name: { type: 'string' },
                type: { type: 'string', enum: ['file', 'directory'] },
                size: { type: 'number' },
              },
            },
          },
        },
      },
      render: (_args, value) => {
        const v = value as {
          path: string
          entries: Array<{ name: string; type: string; size: number }>
        }
        const lines = [`Directory: ${v.path}`, '']
        for (const entry of v.entries) {
          const sizeStr = entry.type === 'file' ? ` (${entry.size} bytes)` : '/'
          lines.push(`  ${entry.name}${sizeStr}`)
        }
        return [{ type: 'text', text: lines.join('\n') }]
      },
    },
    isConcurrencySafe: () => true,
    async execute(args, exec) {
      const cwd = exec.agent?.session?.header?.cwd ?? process.cwd()
      const targetPath = args.path ? join(cwd, args.path) : cwd
      const entries = await readdir(targetPath, { withFileTypes: true })
      const result = await Promise.all(
        entries.map(async (entry) => {
          const fullPath = join(targetPath, entry.name)
          const stats = entry.isFile() ? await stat(fullPath) : null
          return {
            name: entry.name,
            type: entry.isDirectory() ? 'directory' : 'file',
            size: stats?.size ?? 0,
          }
        })
      )
      return {
        path: targetPath,
        entries: result.sort((a, b) => {
          if (a.type !== b.type) return a.type === 'directory' ? -1 : 1
          return a.name.localeCompare(b.name)
        }),
      }
    },
  }))

  // ── 3. Permission interception: require confirmation for dangerous tools ──
  ctx.on('tools/pre-execute', async (exec, next) => {
    const isDangerous = exec.name.includes('delete') || exec.name.includes('rm')
    if (!isDangerous) return next()
    return {
      kind: 'ask',
      reason: `About to run: ${exec.name}(${exec.arguments})`,
    }
  })

  // ── 4. Execution time metering ─────────────────────────────────
  ctx.on('tools/execute', async (exec, next) => {
    const start = performance.now()
    const result = await next()
    const elapsed = Math.round(performance.now() - start)
    console.log(`[${exec.name}] ${result.isError ? 'ERROR' : 'OK'} ${elapsed}ms`)
    return result
  })

  // ── 5. Session Turn summary observation ───────────────────────
  ctx.on('session/event', (session, event) => {
    if (event.type !== 'turn/end') return
    const reason = event.data.reason.kind
    const turnNum = event.data.turn
    console.log(`[Turn ${turnNum}] ${reason}`)
  })

  // ── Note ───────────────────────────────────────────────────────
  // No cleanup code anywhere in this file.
  // Every registration made through ctx (tools.register, systemPrompt.section, ctx.on)
  // is tracked by Cordis and automatically disposed when the plugin unloads.
}

The whole plugin is about 90 lines including comments. Five concerns, 10-20 lines each. This is roughly the right size for a typical dsh plugin.

Adding the Plugin to a Bundle

Once the plugin is written, it needs to be wired into a Bundle:

// bundle.ts (pseudocode: Bundle configuration file)
import workspaceContext from './packages/workspace-context/src/index.ts'

export default defineBundle([
  // Core service providers must be listed before the plugins that depend on them.
  // workspace-context declares inject: ['tools', 'systemPrompt'],
  // so both providers must appear before it.
  coreToolsPlugin,          // provides the tools service
  systemPromptPlugin,       // provides the systemPrompt service

  workspaceContext,         // our plugin

  // Other business plugins...
])

Cordis handles service readiness ordering based on inject declarations — even if the order in the Bundle is wrong, Cordis will wait for dependencies to be ready before activating the plugin, rather than crashing on startup.

Series Recap: Which Articles This Plugin Uses

This 90-line plugin covers most of the core mechanisms from the series:

Plugin feature	Series article
`ctx.tools.register()` — register a tool	Article 03: Tool System
`defineTool` + `isConcurrencySafe` — concurrency flag	Article 03: Tool System
`tools/pre-execute` — permission interception	Article 03: Tool System
`tools/execute` — wrap execution	Article 03: Tool System
`ctx.systemPrompt.section()` — inject a prompt section	Article 06: System Prompt Assembly
`session/event` — Turn-end observation	Article 09: Observability
`export const inject` — Cordis service dependencies	Article 02: Cordis Plugin System
`ctx` lifecycle auto-dispose	Article 02: Cordis Plugin System
`session.header.cwd` — Session context	Article 05: Session Memory

What this plugin doesn't use: multi-Agent collaboration (article 08) and capability seams (article 07). Those are higher-level mechanisms, typically configured at the framework level rather than touched by individual business plugins.

The dsh Design Philosophy

This is a good place to close out the series.

dsh's core design idea is simple: everything is a plugin, everything has a boundary.

You don't need to fork the core to change almost anything about behavior — adding tools, modifying prompts, intercepting dangerous operations, wiring up monitoring — all of that happens through plugins. Boundaries are enforced by Cordis's ctx: a plugin only affects what it registers, and when it unloads, it leaves no trace.

This pattern comes from Cordis. dsh applies it to an Agent runtime.

A production-grade Agent isn't just 'model + tool calls'. It needs permission controls, observability, persistence, multi-Agent coordination — it needs stable handles for all those engineering concerns. dsh's architecture is specifically designed to turn those problems into composable, swappable, testable pieces.

This is still an early field. Many best practices are still forming, many edge cases don't have standard answers yet. But with this architectural foundation, you at least know where to add code and where to look when something goes wrong.

Series Index

If you're just starting, work through these in order:

Article 01: What is dsh and why use it
Article 02: The Cordis plugin system — plugins, services, and lifecycles
Article 03: The tool system — registration, execution, interception
Article 04: The Agent loop — how a single conversation run works
Article 05: Session memory — how state is stored and read
Article 06: System prompt assembly — how prompts are constructed
Article 07: Capability seams — swappable Agent capabilities
Article 08: Multi-Agent collaboration — sub-Agents and task dispatch
Article 09: Observability — token metering and telemetry
Article 10: Complete plugin walkthrough (this article)

Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.

Find more on my Homepage

DeepSeek Harness Series (09): Observability — How to Know What Your Agent Is Actually Doing

WonderLab — Thu, 17 Sep 2026 10:04:56 +0000

Start with a Frustrating Scenario

Your Agent runs for three minutes, then throws an error.

You stare at the logs — nothing useful. You don't know which tools it called, where it got stuck, or where the tokens went. You certainly don't know why it failed.

That's what missing observability looks like in practice.

Why Observability Matters More for Agents

With a traditional service, you can add breakpoints and read stack traces. Agents are different.

Agents are non-deterministic. The same input might produce a completely different sequence of tool calls. You can't set a breakpoint inside "the model's decision-making" — that's a black box.

Debugging requires log reconstruction. The model's 'reasoning' only manifests in the text and tool calls it outputs. You need to capture all of that so you can rebuild the inference chain after the fact.

Token costs are opaque. Across a multi-turn Agent conversation, which step is the most expensive? Is it the long tool result in turn three? The oversized system prompt? Without metering, you can't even identify where to optimize.

Production incidents need evidence. When a user reports a bad answer, you need to reconstruct the full execution chain from that moment — which tools ran, what they returned, what the model saw.

Session Logs: The Most Complete Observability Data Source

Recall from article 05: a dsh Session is an append-only log of typed events.

That design isn't only about persistence — the log itself is the most complete observability source you have.

Every Session event contains:

Structure of every SessionEvent:
  - type:  event type (e.g. 'tool/call', 'turn/end', 'assistant/message')
  - seq:   monotonically increasing sequence number (starts at 0)
  - time:  Unix timestamp in milliseconds
  - data:  typed event payload (structure varies by type)

This means: as long as you can read the Session log, you can reconstruct the entire execution — what happened at each step, how long it took, whether anything failed.

Real-Time Listening: session/event

You don't have to wait for the log to finish before analyzing it. dsh provides the session/event event, which fires for every entry as it's written:

// Listen to all events on a Session in real time
ctx.on('session/event', (session, event) => {
  // This callback fires for every event
  console.log(`[${event.type}] seq=${event.seq} time=${event.time}`)

  // Check for tool calls
  if (event.type === 'tool/call') {
    // event.data.name is the tool name
    // event.data.arguments is a raw JSON string (model output, not yet parsed)
    // event.data.callId is the unique ID for this invocation
    console.log(`  Tool: ${event.data.name}`)
    console.log(`  Args: ${event.data.arguments}`)
  }

  // Check for tool execution results
  if (event.type === 'tool/result') {
    // Detect failure by looking for isError: true in the content blocks
    const isError = event.data.message.content.some(
      b => b.type === 'tool_result' && b.isError
    )
    console.log(`  Result: ${isError ? 'ERROR' : 'OK'}`)
  }
})

During development, this listener is invaluable — you see what the Agent is doing in real time, not just after it finishes.

Token Metering: ctx.tokenMeter

Knowing 'what happened' is step one. Knowing 'how much it cost' is equally important.

dsh provides ctx.tokenMeter to measure the token pressure of the current Session:

// TokenMeasurement interface (from packages/llm/token-meter/src/types.ts)
interface TokenMeasurement {
  // How many events this measurement consumed (for caching, avoids recomputation)
  readonly logRevision: SessionLogOffset

  // Total token pressure for the current request (input + output combined)
  readonly totalTokens: number

  // Token count of the current surface (history visible to the model)
  readonly surfaceTokens: number

  // Signed delta of surface tokens vs. the last successful request
  // (can be negative if context shrank)
  readonly surfaceDeltaTokens: number

  // Surface nodes in positional order with their individual token counts
  // (lets you see how many tokens each message contributes)
  readonly nodes: readonly TokenSurfaceNode[]
}

Key concepts:

surfaceTokens: How many tokens the model actually sees as input for this request. This drives the 'input token' portion of your API bill.
totalTokens: Input + output combined — the full cost of the request.
surfaceDeltaTokens: How much the surface grew since the last request. If this keeps increasing, your context is ballooning and you might need a compression strategy.

Usage example:

// Print a token summary at the end of each Turn
ctx.on('session/event', (session, event) => {
  // Only care about Turn-end events
  if (event.type !== 'turn/end') return

  // Call measure to get the current Token measurement for this Session
  const measurement = ctx.tokenMeter.measure(session)

  console.log(`Turn ${event.data.turn} ended:`)
  console.log(`  Surface tokens: ${measurement.surfaceTokens}`)
  console.log(`  Total tokens:   ${measurement.totalTokens}`)

  // Show delta — positive means context is growing
  const delta = measurement.surfaceDeltaTokens
  const sign = delta > 0 ? '+' : ''
  console.log(`  Delta:          ${sign}${delta}`)

  // Warn if context is growing too fast
  if (delta > 2000) {
    console.warn('  ⚠ Context growing fast, consider compression')
  }
})

The Telemetry Seam: ctx.sessionTelemetry

session/event listeners are great for development, but in production you need to ship data to external systems — Grafana, Datadog, CloudWatch, and so on.

dsh provides a 'Telemetry Seam' for this: ctx.sessionTelemetry.

The word 'seam' is deliberate — it's a standardized interface that lets you plug in any telemetry backend, while the harness itself stays independent of any particular monitoring system.

Structure of each telemetry record:

// SessionTelemetryRecord (from packages/session/session-telemetry/src)
interface SessionTelemetryRecord {
  // Two channels:
  //   'ledger': full mirror of Session log events, one-to-one mapping
  //   'ops':    operational signals, only emitted for special situations
  channel: 'ledger' | 'ops'

  // Timestamp in milliseconds
  time: number

  // Severity level
  severity: 'info' | 'warn' | 'error'

  // Identifying attributes (for querying and filtering)
  // e.g. session.id, event.type, event.seq
  attributes: Record<string, string | number>

  // Full payload: a deep copy of event.data
  body: unknown
}

What each channel does

ledger channel: A complete mirror of the Session log. Every Session event produces one corresponding ledger record. This is your source of truth for audit and replay.

Includes:

Every assistant/message (with full streaming data)
Every tool/call and tool/result
Failed assistant/attempt events (things the model generated but didn't commit)
All lifecycle events: turn/start, turn/end, etc.

ops channel: Operational signals, just two kinds:

agent-error: The Agent failed outside of a Turn (e.g. initialization error)
shutdown: The Agent shut down cleanly

How severity is determined

error: tool results with isError: true, turn/end with an error reason, agent-error ops events
Everything else: info

This mapping means you can filter severity === 'error' in your monitoring system to see all failures instantly, without writing custom classification logic.

OpenTelemetry Integration

dsh provides an official OTel Provider plugin: dsh-session-telemetry-otel.

How to wire it in (conceptual):

// Add this plugin to your Bundle configuration (pseudocode)
// This connects ctx.sessionTelemetry to the OTel backend
'@deepseek-ai/dsh-session-telemetry-otel'

// Internally, the plugin will:
// 1. Register an OTel backend implementation for ctx.sessionTelemetry
// 2. Send each SessionTelemetryRecord via the OTel JS SDK's Logger API
// 3. Support configurable Exporters (OTLP, Console, File, etc.)

A few design principles worth knowing:

Boundary axiom: The harness only calls emit(). Batching, retries, and queuing are the OTel SDK's responsibility — the harness doesn't touch them. Both sides can evolve independently.

Best-effort delivery: Telemetry records may be duplicated or lost. Consumers should deduplicate ledger records using the (session.id, format_version, event.seq) tuple, rather than assuming exactly-once delivery.

flush is optional: You can call flush() at the end of each Turn, but the OTel backend doesn't implement it by default (to avoid concurrency conflicts). If you need strong consistency, configure it yourself.

Hands-On: A Simple Debug Observer Plugin

Let's combine everything above into a complete debug observability plugin:

// debug-observer.ts — observability plugin for development use
// Usage: add to your Bundle during development;
//        in production, swap in dsh-session-telemetry-otel instead

export const name = 'debug-observer'

// Declare dependency injection for tokenMeter
export const inject = ['tokenMeter']

export function apply(ctx: Context): void {
  // ── 1. Watch tool calls ────────────────────────────────────
  ctx.on('session/event', (session, event) => {
    if (event.type !== 'tool/call') return

    console.log(`[Tool Call] ${event.data.name}`)
    console.log(`  Call ID: ${event.data.callId}`)
    // arguments is a raw JSON string — straight from the model, not yet parsed
    console.log(`  Args: ${event.data.arguments}`)
  })

  // ── 2. Watch tool results ──────────────────────────────────
  ctx.on('session/event', (session, event) => {
    if (event.type !== 'tool/result') return

    const blocks = event.data.message.content
    const isError = blocks.some(b => b.type === 'tool_result' && b.isError)
    const icon = isError ? '✗' : '✓'

    // toolUseId links this result back to its matching tool/call event
    const toolUseId = blocks[0]?.toolUseId ?? 'unknown'
    console.log(`[Tool Result] ${icon} (call: ${toolUseId})`)
  })

  // ── 3. Print token summary at each Turn end ────────────────
  ctx.on('session/event', (session, event) => {
    if (event.type !== 'turn/end') return

    // reason.kind can be 'complete', 'error', 'interrupted', etc.
    const reason = event.data.reason.kind
    const measurement = ctx.tokenMeter.measure(session)

    console.log(`\n[Turn ${event.data.turn}] ended: ${reason}`)
    console.log(`  Surface: ${measurement.surfaceTokens} tokens`)
    console.log(`  Total:   ${measurement.totalTokens} tokens`)

    const delta = measurement.surfaceDeltaTokens
    const sign = delta > 0 ? '+' : ''
    console.log(`  Delta:   ${sign}${delta}`)

    // If the Turn ended with an error, print the full reason object
    if (reason === 'error') {
      console.error(`  Error: ${JSON.stringify(event.data.reason)}`)
    }
  })

  // ── 4. Watch Session lifecycle ─────────────────────────────
  ctx.on('session/created', (session) => {
    console.log(`\n[Session] created: ${session.id}`)
  })

  ctx.on('session/disposed', (session) => {
    console.log(`[Session] disposed: ${session.id}`)
  })
}

This plugin drops into your Bundle during development and gives you a complete trace of everything the Agent does. In production, swap it for the OTel plugin to route that same data into your monitoring stack.

Debugging Tip: Reading JSONL Log Files

dsh persists Session logs as JSONL files — one JSON object per line, each representing one SessionEvent.

Some useful command-line analysis recipes:

# See all tool calls (extract just the tool names)
cat session.jsonl | grep '"type":"tool/call"' | jq '.data.name'

# See failed assistant attempts (model output that wasn't committed)
cat session.jsonl | grep '"type":"assistant/attempt"' | jq '.'

# Summarize token usage per turn (from assistant/message usage field)
cat session.jsonl | grep '"type":"assistant/message"' | jq '.data.usage'

# See how each Turn ended (complete vs. error)
cat session.jsonl | grep '"type":"turn/end"' | jq '.data.reason.kind'

# Find any failed tool executions
cat session.jsonl | grep '"type":"tool/result"' | jq 'select(.data.message.content[].isError == true)'

These assume you have jq installed. On Windows, PowerShell's ConvertFrom-Json provides equivalent functionality.

The Four Layers of Observability

Real-time observation (development)
  └─ session/event listener → each event printed to console immediately

Audit and replay (post-mortem)
  └─ JSONL log file → full execution chain reconstruction, analyzable with jq

Token usage analysis
  └─ ctx.tokenMeter.measure(session) → per-node token counts across the surface
     → pinpoint which messages are burning your budget

Production monitoring (systems level)
  └─ ctx.sessionTelemetry + OTel plugin → ship to Grafana / Datadog / CloudWatch
     → alerts, dashboards, and error tracking all wired up

Wrapping Up

Observability isn't a nice-to-have for Agents — it's the only way to debug them.

dsh is designed with this in mind: Sessions are event logs, and event logs are audit data by definition. ctx.tokenMeter makes token consumption transparent. ctx.sessionTelemetry provides a standardized seam so you can choose any backend.

The core pattern is simple: session/event listeners → JSONL persistence → Telemetry Seam → OTel backend. Which layer you use depends on your context, but all of them can run simultaneously without interfering with each other.

Next up is the final article in the series. We'll bring everything together — tool registration, Session management, error handling, and observability — and build a complete, production-quality plugin from scratch.

Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.

Find more on my Homepage

One Open Source Project a Day (No. 184): WeKnora — Tencent's Enterprise Knowledge Framework, From RAG Q&A to a Self-Evolving Wiki

WonderLab — Thu, 17 Sep 2026 10:04:00 +0000

Introduction

"A document shouldn't just be retrievable — it should be understood, organized, and continuously maintained."

This is the 184th article in the "One Open Source Project a Day" series. Today's project is WeKnora.

Most enterprise knowledge base products today still stop at the standard RAG pipeline: upload a document → vectorize it → similarity search → stuff results into a prompt. That pipeline handles simple questions fine, but falls apart when the task requires multi-step reasoning, synthesis across multiple documents, or something like "figure out what this document actually says and organize it into something I can browse like an encyclopedia."

WeKnora is Tencent's answer to that gap. It isn't another RAG framework — it stacks three capabilities together: fast Q&A via RAG, autonomous multi-step reasoning via a ReAct agent that orchestrates tools on its own, and long-term knowledge accumulation via a Wiki mode that auto-generates an interlinked knowledge graph. It's the core technology framework behind the WeChat Dialog Open Platform, meaning it's already been through a round of real enterprise deployment.

25.4k Stars, MIT License, Go backend.

What You Will Learn

WeKnora's three core capabilities: RAG Q&A, ReAct agent, and Wiki mode
The modular pipeline architecture: document parsing → vectorization → retrieval → inference
The design behind its cross-session long-term memory mechanism
What the 29 tools in its official MCP Server can do
Enterprise deployment considerations: multi-workspace RBAC, security encryption, observability

Prerequisites

Basic understanding of RAG (Retrieval-Augmented Generation)
Familiarity with Agent tool-calling and the MCP protocol
Optional: knowledge of vector databases (pgvector/Milvus, etc.)

Project Background

What It Is

WeKnora's official positioning: "an open-source, LLM-powered knowledge framework built for enterprise-grade document understanding, semantic retrieval, and autonomous reasoning."

The core problem it's solving: turning scattered documents into a queryable, reasoning-capable, continuously evolving knowledge asset. Those three adjectives map directly to its three capabilities — queryable (RAG), reasoning-capable (ReAct agent), continuously evolving (Wiki mode's self-maintaining version control).

Team and Background

Organization: Tencent
Website: weknora.weixin.qq.com
Related product: Core technology framework of the WeChat Dialog Open Platform
License: MIT License

Project Stats

⭐ GitHub Stars: 25,400+
🍴 Forks: 3,500+
👀 Watchers: 118
📦 Current version: 0.8.0
💻 Primary language: Go (backend)
📄 License: MIT

What It Does

The Problem It Solves

Traditional enterprise knowledge base (pure RAG):
  Upload document → chunk → vectorize → similarity search → prompt stuffing → answer
  ↑ Handles "what does this document say about X" fine
  ↑ Struggles with "synthesize a trend analysis across three documents"
  ↑ The knowledge base never organizes itself — it stays a pile of raw documents forever

WeKnora (three-in-one):
  Simple query   → RAG fast Q&A (low latency, good enough)
  Complex task   → ReAct agent
                   ├── decides autonomously whether to retrieve
                   ├── calls MCP tools / web search / sandboxed code execution
                   └── returns a synthesized answer after multi-step reasoning
  Long-term      → Wiki mode
                   ├── agent distills raw documents into an interlinked Markdown knowledge base
                   ├── generates an interactive knowledge graph
                   └── supports manual editing, version history, and one-click rollback

Use Cases

Enterprise internal knowledge Q&A
- Centralize employee handbooks, product docs, and technical specs; employees query directly via WeCom/Feishu
Customer service / pre-sales auto-response
- Connect IM channels (Slack/Telegram/WeCom), turn product documentation into an auto-answering knowledge base
Automating complex research tasks
- The ReAct agent calls web search + internal document retrieval + code sandbox to complete multi-step "research → analyze → produce a report" tasks
Continuous knowledge asset maintenance
- Wiki mode lets the document knowledge base self-update and interlink like an encyclopedia, instead of staying a pile of isolated PDFs
Zero-code Q&A deployment
- Through the WeChat Dialog Open Platform, non-technical staff can configure a working knowledge Q&A system

Quick Start

# Prerequisites: Docker, Docker Compose, Git

git clone https://github.com/Tencent/WeKnora.git
cd WeKnora
cp .env.example .env
docker compose pull
docker compose up -d

# Visit http://localhost after startup

Optional feature modules (Profiles):

# Enable knowledge graph (Neo4j)
docker compose --profile neo4j up -d

# Enable object storage (MinIO)
docker compose --profile minio up -d

# Enable observability tracing (Langfuse)
docker compose --profile langfuse up -d

# Enable all features
docker compose --profile full up -d

If using a local Ollama model, run ollama serve first. To upgrade, set WEKNORA_VERSION in .env, then run docker compose pull && docker compose up -d.

Core Features

1. Three Core Capabilities

Capability	Best For	Key Mechanism
RAG fast Q&A	Everyday simple queries	Vector retrieval + direct generation
ReAct agent	Complex multi-step tasks	Autonomous orchestration of retrieval/tools/sandbox/search
Wiki mode	Long-term knowledge accumulation	Auto-distillation + interlinking + version control

2. Cross-Session Long-Term Memory

Unlike most Q&A systems that start from zero every conversation, WeKnora maintains memory across sessions along several dimensions:

Profile: user identity and background
Preferences: historical choice patterns
Facts: specific facts confirmed during conversation
Tasks: multi-turn tasks being tracked
Interests: topic areas the user cares about

3. Official MCP Server: 29 Tools

WeKnora ships an official MCP Server that exposes its own capabilities as standard MCP tools, letting Claude Code, Cursor, and other AI tools call WeKnora's knowledge base functions directly — retrieval, writing, Wiki editing, all reachable via MCP.

4. Multi-Source Data Ingestion

Source Type	Specific Support
Enterprise collaboration tools	Feishu, Tencent IMA, DingTalk Docs, Yuque
Dev platforms	GitLab
General platforms	Notion, RSS
Document formats	PDF, Word, images, Excel, XMind, and 10+ other formats

5. 20+ LLM Provider Support

Covers major international providers (OpenAI, Azure OpenAI, Anthropic Claude, Gemini) and major Chinese providers (DeepSeek, Qwen, Zhipu, Hunyuan), plus LiteLLM and Ollama as unified access layers.

6. Enterprise Security and Permissions

Multi-workspace RBAC: four-tier role matrix with fine-grained permission control
Encryption: AES-256-GCM data encryption
Transport security: gRPC TLS
SSRF protection: guards against server-side request forgery
Scoped API keys: limit the access range of any given key

A Deeper Look

The Modular Pipeline Architecture

WeKnora's architecture emphasizes that every stage is swappable:

Document Parsing
   ↓ parsers for PDF / Word / Excel / images / XMind and more
Vectorization
   ↓ supports multiple embedding models
Retrieval
   ↓ supports PostgreSQL(pgvector) / Elasticsearch / OpenSearch / Milvus / Weaviate / Qdrant
LLM Inference
   ↓ supports 20+ LLM providers, unified through LiteLLM

The benefit of this design: enterprises can swap in whatever infrastructure they already have. Teams already using Milvus don't need to migrate their vector store; teams with a privately deployed model can plug it in directly. This is also the technical foundation of the "data sovereignty" promise — the entire pipeline can run in a closed loop entirely within a private environment, with no data ever needing to leave the corporate network.

The Design Logic Behind RAG, ReAct, and Wiki

Why not build one "universal" mode instead of splitting into three?

Pure RAG mode's limitation:
  Retrieve → prompt → generate
  ↑ Low latency, low cost, but weak reasoning
  ↑ Good for "does this document mention X"-type questions

Pure Agent mode's problem:
  Every query walks the full reasoning chain
  ↑ Even simple questions trigger multi-round tool calls — high latency, high cost
  ↑ Users perceive the system as "slower"

WeKnora's layered design:
  Simple query   → straight to RAG (fast)
  Complex query  → escalates to ReAct (accurate)
  Knowledge accumulation → runs asynchronously via Wiki mode (doesn't affect real-time Q&A)

This "escalate complexity on demand" design is fundamentally a dynamic trade-off between latency/cost and capability — not every question needs agent-level reasoning.

Wiki Mode: From a Pile of Documents to a Self-Maintaining Knowledge Base

Wiki mode is WeKnora's most distinctive capability. Most RAG systems treat documents as raw material for retrieval — once retrieval is done, the documents themselves never change.

WeKnora's Wiki mode flips this: the agent actively reads raw documents, distills them into structured Markdown pages, and interlinks those pages into a Wikipedia-like knowledge network. That network:

Has an interactive knowledge graph visualization
Supports manual editing (what the agent generates isn't final — it's a draft)
Has complete version history (Git-like diff comparison)
Supports one-click rollback to any historical version

This solves a common RAG pain point: when raw documents are low-quality (disorganized structure, duplicated information, stale content mixed with current), what gets retrieved is also low-quality. Wiki mode essentially has the AI do a round of "document cleanup" first, so subsequent Q&A is built on organized knowledge rather than raw mess.

How It Compares to Similar Projects

Dimension	LangChain + Custom RAG	Dify	RAGFlow	WeKnora
Out-of-the-box	❌ Requires heavy custom dev	✅	✅	✅
ReAct agent	Must build yourself	Limited	Limited	✅ Native
Self-maintaining Wiki knowledge base	❌	❌	❌	✅
Cross-session long-term memory	Must implement yourself	Limited	❌	✅
Official MCP Server	❌	Partial	❌	✅ 29 tools
Enterprise RBAC	Must implement yourself	Yes	Limited	✅ Four-tier roles
Chinese LLM ecosystem support	Must integrate yourself	Average	Average	✅ Native coverage

WeKnora's differentiation lies in its native Wiki mode and ReAct agent integration, plus native support for the Chinese LLM ecosystem (DeepSeek, Qwen, Hunyuan) and Chinese enterprise collaboration tools (Feishu, DingTalk, WeCom) — a clear advantage for deployment in Chinese enterprises.

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/Tencent/WeKnora
🌐 Website: weknora.weixin.qq.com
📄 License: MIT License
🐛 Issues: GitHub Issues

Related Resources

Model Context Protocol — The standard protocol underlying WeKnora's official MCP Server
Langfuse — The observability/tracing tool WeKnora integrates with
pgvector — One of WeKnora's supported vector retrieval backends

Summary

Key Takeaways

A three-in-one capability matrix: RAG for simple queries, a ReAct agent for complex tasks, Wiki mode for long-term knowledge — escalating complexity on demand
Wiki mode is the standout feature: distilling raw documents into a self-maintaining, rollback-capable, interlinked knowledge base, rather than leaving documents permanently "raw"
A modular pipeline: every stage — parsing, vectorization, retrieval, inference — is swappable to fit existing enterprise infrastructure
An official MCP Server: 29 tools that let Claude Code and other AI tools call WeKnora's capabilities directly
Native support for the Chinese ecosystem: DeepSeek/Qwen/Hunyuan plus Feishu/DingTalk/WeCom — deeply localized

Who This Is For

Enterprise IT / knowledge management teams: need to turn scattered documents into a maintainable knowledge asset, not just "something searchable"
Developers building for the Chinese AI ecosystem: need a knowledge framework with native support for domestic LLMs and enterprise collaboration tools
Teams building complex agent applications: need ReAct-level multi-step reasoning rather than settling for plain RAG
Organizations prioritizing data sovereignty: the entire pipeline can be deployed privately with no data leaving the internal network

One-Line Verdict

WeKnora is trying to solve this: a knowledge base shouldn't just be a pile of retrievable documents — it should be a living knowledge system that organizes itself, reasons, and keeps evolving.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 183): Openship — Vercel-Level Deploy Experience on Your Own Server

WonderLab — Wed, 16 Sep 2026 05:24:34 +0000

Introduction

"Deploy anything. Own everything."

This is the 183rd article in the "One Open Source Project a Day" series. Today's project is Openship.

Vercel's developer experience is genuinely excellent — push code, and a few minutes later your app is live with SSL, routing, and previews handled automatically. But it has one fundamental problem: your application runs on someone else's servers, pricing is entirely up to the vendor, a traffic spike can blow up your bill, and if you want to leave you have to migrate everything out.

The alternative is running Docker yourself — full control, but SSL, CI/CD, domain routing, databases — everything needs manual setup. High floor.

Openship sits between the two: Vercel-level operating experience, but the application runs on your own server. Push code, it builds a Docker image locally, transfers it over SSH, and OpenResty takes over routing and TLS. Your server, your data, your bill.

12.3k Stars, Apache 2.0, completely free to self-host.

What You Will Learn

Openship's three deployment modes (desktop app / self-hosted server / Openship Cloud)
The "build locally, deploy remotely" architecture and why it matters
How Openship positions itself against Vercel/Netlify and Coolify/Dokploy
The built-in services bundle: databases, mail server, object storage, backups
Why "containers keep running after you delete a project" is a significant design decision

Prerequisites

Basic understanding of Docker
Some experience deploying a web application
Optional: familiarity with SSH and reverse proxies

Project Background

What It Is

Openship is an open-source, self-hosted PaaS (Platform as a Service). It replicates the operational experience of commercial PaaS platforms (Vercel, Railway, Render) on your own infrastructure.

The core promise is no vendor lock-in: Openship is built on open standards — Docker, OCI images, S3 protocol, SMTP, ACME (the Let's Encrypt certificate automation protocol). If you ever want to walk away from Openship, your containers, databases, and images are all in standard formats and can be migrated anywhere.

One particularly notable design: when you delete an Openship project, the containers on your server keep running. This is the opposite of most PaaS platforms, where deleting a project makes your deployment disappear.

Author / Team

Author: oblien (GitHub: @oblien)
Website: openship.io
License: Apache License 2.0

Project Stats

⭐ GitHub Stars: 12,300+
🍴 Forks: 1,100+
📄 License: Apache 2.0 (free including commercial use)
🌐 Website: openship.io

What It Does

The Problem It Solves

Option A: Commercial PaaS (Vercel / Netlify / Railway)
  ✅ Push-to-deploy, excellent experience
  ✅ SSL / routing / CI fully automated
  ❌ App runs on vendor servers — no data sovereignty
  ❌ Traffic spikes = bill spikes
  ❌ Migrating away means moving everything

Option B: Roll your own Docker + Nginx + CI stack
  ✅ Full control, data stays with you
  ❌ SSL certs, reverse proxy, CI/CD, databases — all manual
  ❌ Everything needs individual maintenance

Option C: Openship
  ✅ Push-to-deploy (same experience as Vercel)
  ✅ SSL / routing / databases / email all built in
  ✅ App runs on your server
  ✅ Built on open standards — walk away anytime
  ✅ Self-hosting is completely free

Three Ways to Run It

Mode 1: Desktop App (solo developers)

Download the macOS / Windows / Linux desktop client. The control plane runs locally. It manages remote servers over SSH — no agent installation on the server, no control console exposed to the public internet.

Best for: personal projects, side projects, not wanting to maintain extra infrastructure.

Mode 2: Self-Hosted Server (teams)

curl -fsSL https://get.openship.io | sh
openship   # interactive wizard walks through setup

Run the control plane on a server. Team members collaborate through the web dashboard.

Mode 3: Openship Cloud (zero maintenance)

Managed service starting at $10/month. You don't maintain the control plane, but the deployment target can still be your own servers (hybrid mode).

The Deployment Flow

1. Connect your repository (GitHub / GitLab / Bitbucket)

2. Push code
   ↓
3. Openship detects your stack (Node / Python / Go / …)
   ↓
4. Builds a Docker image on the local machine (not the production server)
   ↓
5. Transfers the image to the target server over SSH
   ↓
6. Container starts on the server
   ↓
7. OpenResty takes over routing + Let's Encrypt provisions SSL automatically
   ↓
App is live ✓

The key design point: building happens on the machine running the Openship console, not on the production server. Production server resources aren't consumed by the build process, and a build failure doesn't affect anything running live.

Supported Stacks

Language / Runtime	Databases	Other Services
Node.js	PostgreSQL	Object storage (S3-compatible)
Python	MySQL	Built-in mail server (SMTP)
Go	MongoDB	Scheduled backups
Rust	Redis	Let's Encrypt SSL
PHP		Custom domains
Ruby		Private networking
Java		DDoS protection
.NET

The Built-In Services Bundle

This is where Openship pulls ahead of many comparable tools — no third-party service integrations required:

Built-in mail server: Full SMTP service with DKIM/SPF/DMARC support. No Mailgun or SendGrid subscription needed.

Built-in databases: Postgres, MySQL, MongoDB, Redis — one-click creation with automatic backups.

Scheduled backups: Automatic backups on a schedule, with one-click restore.

SSL certificates: Let's Encrypt automatic provisioning and renewal, with wildcard domain support.

Interfaces

Interface	Best For
Desktop app (Mac/Windows)	Solo developers, local control
Web dashboard	Team collaboration, browser access
CLI	Scripting, DevOps pipelines
REST API	Programmatic integration
MCP endpoint	AI agent calls (Claude Code, etc.)

A Deeper Look

The Value of "Build Locally, Ship Remotely"

Most self-hosted deployment platforms (Coolify, Dokploy, Caprover, etc.) build on the target server: pull code to the server, run the build process on the server.

Openship inverts this: builds run on the machine hosting the Openship console, and the resulting Docker image is transferred via SSH to the target server.

Coolify / Dokploy model:
  Code → target server (pull + build + run)
  ↑ Build process competes with live services for CPU/RAM
  ↑ Build failure can destabilize the server running production

Openship model:
  Code → console machine (build) → SSH transfer → target server (run only)
  ↑ Build and runtime are physically separated
  ↑ Target server resources are 100% for live traffic
  ↑ Build failures don't touch production

On a resource-constrained VPS (say, a 2-core 4GB Hetzner box), this architectural difference is significant — the same hardware delivers more stable live service performance with Openship's model.

The OpenResty Routing Layer

Openship uses OpenResty (Nginx + Lua extensions) as its edge proxy, handling three things:

HTTP routing: mapping domains to the correct containers
TLS termination: Let's Encrypt certificate provisioning, renewal, and HTTPS handling
Zero-downtime cutover: traffic only shifts from old to new container once the new one is healthy

There's a notable design detail about the sequencing of TLS and routing: routing and TLS take over after the application container is already running. This means if DNS resolution or certificate provisioning encounters an issue, it surfaces in the console as "action required" rather than failing the entire deployment and rolling back — which makes root-cause debugging far easier.

Competitive Positioning

Where Openship sits on the landscape:

Commercial PaaS (Vercel / Railway / Render)
  ↑ Easiest experience, most expensive, no data sovereignty

Self-hosted PaaS (Coolify / Dokploy / Caprover)
  ↑ Self-controlled, free, but fewer built-ins, no desktop client

Openship
  ↑ Among self-hosted PaaS options: richest built-in services
     (mail server, backups, object storage)
  ↑ The only self-hosted PaaS with a desktop client
  ↑ The only self-hosted PaaS with an MCP endpoint
  ↑ Local-build architecture

Raw Docker + Nginx + CI DIY
  ↑ Most flexible, highest floor

Dimension	Vercel	Coolify	Dokploy	Openship
Data sovereignty	❌	✅	✅	✅
Push-to-deploy	✅	✅	✅	✅
Built-in mail server	❌	❌	❌	✅
Desktop client	❌	❌	❌	✅
MCP endpoint	❌	❌	❌	✅
Local-build architecture	❌	❌	❌	✅
Containers survive project deletion	❌	❌	❌	✅
Free to self-host	—	✅	✅	✅
Stars	—	40k+	15k+	12.3k

Why "Containers Survive Project Deletion" Matters

This design decision is a direct expression of Openship's core philosophy: the platform is a tool, not a chain.

On Vercel, your deployment lifecycle is bound to the Vercel platform — delete a project, the deployment is gone, with no separation between them.

Openship's logic: your containers run on your server. Openship manages them; it doesn't own them. Delete the project configuration in Openship, and the containers on your server keep running. You can operate them with standard Docker commands, or re-import them into Openship management later.

This makes migrating or switching platforms nearly risk-free — the worst-case scenario is "back to manual management," not "service disappears."

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/oblien/openship
🌐 Website: https://openship.io
📚 Docs: https://openship.io/docs
🐛 Issues: GitHub Issues

Related Resources

Coolify — 40k Stars, the most feature-complete self-hosted PaaS alternative
Dokploy — 15k Stars, lightweight self-hosted deployment platform
OpenResty — The Nginx + Lua platform powering Openship's routing layer

Summary

Key Takeaways

Local-build architecture: builds happen on the control console machine; the production server only runs containers — resources are separated
Built-in services bundle: databases, mail server, object storage, backups — no third-party service subscriptions required
Desktop client: solo developers can manage remote deployments over SSH without deploying an additional server
Delete = detach, not delete = destroy: the platform manages your containers; it doesn't own them
MCP endpoint: AI agents can call Openship directly to trigger deployment operations

Who This Is For

Developers tired of Vercel bills: want the same experience but with the app running on their own VPS
Home server / VPS tinkerers: want a full PaaS experience instead of pure manual Docker ops
Data-sovereignty-conscious teams: not willing to put build artifacts and databases on a third-party platform
AI application developers: need to let an Agent automatically trigger deployment pipelines via MCP

One-Line Verdict

Openship unpacks a classic contradiction: you want Vercel's experience, but not Vercel's bill or vendor lock-in — it separates the two.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

DeepSeek Harness Series (08): Multi-Agent Collaboration — Subagents and Agent Teams

WonderLab — Wed, 16 Sep 2026 05:23:28 +0000

A Problem One Tool Can't Solve

Imagine this task: a comprehensive refactor of a large codebase — check naming conventions across every file, find all circular dependencies, clean up interface definitions. Over a thousand files.

You could hand this to a single Agent and let it process files one by one. But the problems appear quickly:

Limited context window: by file 300, everything from the beginning has been pushed out
Tools have no reasoning: a tool executes fixed logic — it can't adapt its strategy based on 'which subsystem does this file belong to?'
Tools have no persistent conversation history: a tool finishes and that's it; no state, no memory of what it saw last time

The essence of multi-Agent systems is delegating subtasks to another Agent that has its own independent conversation history. A subagent has its own Session, its own reasoning process, its own tools, and its own context window. The parent Agent only needs to say: 'process this batch of files and tell me what you find.'

The difference from calling a tool is the difference between 'handing something off to another person' and 'starting a machine.'

dsh's Subagent Mechanism

dsh implements Agent delegation via Subagents. There are two ways to launch one:

Model-initiated: when the model decides to delegate a subtask, it calls the built-in tool subagent_spawn
Code-initiated: inside a tool's execute function, call ctx.subagents.start() directly

Launching a subagent requires a SubagentStartRequest. The key fields:

// Simplified from SubagentStartRequest in packages/subagent/subagent/src/types.ts
// The actual type has more fields; these are the most important

interface SubagentStartRequest {
  // The subagent's initial prompt (ContentBlock array — supports text, images, etc.)
  prompt: ContentBlock[]

  // The parent Agent object (provides working directory and lineage info)
  parent: Agent

  // Cancellation signal (lets the parent abort the subagent)
  signal: AbortSignal

  // Optional: restrict which tools the subagent can use
  toolFilter?: ToolRestriction

  // Optional: give the subagent a custom persona (overrides the global deployment persona)
  persona?: string

  // Optional: structured output schema (makes the subagent return a JSON object instead of text)
  outputSchema?: ObjectJsonSchema

  // Optional: max delegation depth (prevents infinite recursion via nested subagents)
  maxDepth?: number
}

A few fields deserve attention:

toolFilter: by default a subagent inherits all of its parent's tools, but in most cases you want to give it a restricted set. A code-analysis subagent only needs to read files — not write them.

persona: a subagent can have its own system prompt prefix. You can give it a specialized role definition so it works within that role rather than inheriting the global persona.

outputSchema: instead of returning plain text, the subagent returns a structured JSON object. The parent Agent receives a type-safe result.structured.

One-Shot vs Continuable

Subagents have two lifecycle modes, distinguished by whether the parent needs to talk to the subagent more than once.

One-shot Subagent:

Launch → subagent completes all its work → returns result → done. One round trip, suitable for independent subtasks.

Continuable Subagent:

After launch, the subagent creates a persistent Session. The parent can keep sending it messages, and the subagent continues executing each time. Suitable for multi-turn collaborative work.

The two modes compared:

One-shot:
  Parent ──start()──► Child executes full task ──result──► Parent
  (subagent's lifecycle ends after completion)

Continuable:
  Parent ──startContinuable()──────► Child Session created, waits for input
  Parent ──sendMessage('batch 1')──► Child processes batch 1 ──result──► Parent
  Parent ──sendMessage('batch 2')──► Child processes batch 2 ──result──► Parent
  Parent ──sendMessage('summarize')► Child summarizes ──final result──► Parent
  (Child Session stays alive throughout, with a continuous conversation history)

The advantage of a continuable subagent: its conversation history is unbroken. It remembers what it processed in earlier turns, and can draw on that memory when summarizing at the end.

Communication Paths: Who Can Message Whom

In a multi-Agent system, a natural question is: can Agents message each other freely?

The answer is no. dsh enforces explicit constraints on communication paths:

Direction	Allowed?	Notes
Parent → Child	✅ Yes	Requires the child's `parentSession` to point to the parent
Child → Parent	✅ Yes	Subagents can message their parent
Sibling → Sibling	❌ No	Two children of the same parent cannot message each other
Grandparent → Grandchild	❌ No	Cross-generation messaging is rejected

       Parent
      /      \
  Child A   Child B
     |
   Grandchild

✅ Parent ──► Child A
✅ Child A ──► Parent
❌ Child A ──► Child B  (sibling messaging rejected)
❌ Parent ──► Grandchild  (cross-generation rejected)

This is a deliberate constraint. Allowing arbitrary messaging turns the Agent network into an untraceable message graph — when debugging, you'd have no idea where a message came from. A clear hierarchy means clear responsibility boundaries.

toolFilter and persona

Tool Filtering

Giving a subagent only the tools it actually needs is the most important access-control mechanism in a multi-Agent system:

// Subagent can only use these three tools.
// Everything else the parent has (write_file, execute_shell, etc.) is invisible to it.
toolFilter: {
  allow: ['read_file', 'search_files', 'list_files'],
  // You can also use deny to exclude specific tools:
  // deny: ['write_file', 'execute_shell']
}

Why bother? It's not only about security — it's about task focus. A code-analysis subagent with write_file in its tool list might 'helpfully' modify code it thinks is wrong. That's not the behavior you want. Give it read-only tools and it can only read.

Persona

// Subagent uses a custom system prompt prefix.
// This overrides the global deployment persona,
// so the subagent operates within a specialized role.
persona: 'You are a specialized test writer. Focus only on writing unit tests for the given code. Do not modify existing source files, do not suggest refactoring.'

The point of persona: the parent Agent can be a general-purpose assistant, while the subagents it spawns are laser-focused experts. Each subagent gets a freshly defined role, unaffected by global configuration.

Structured Output (outputSchema)

Subagents return text by default. But in automated pipelines, the parent Agent usually needs data it can process directly — not a paragraph of natural language.

outputSchema solves this:

// Make the subagent return structured JSON — no text parsing needed (pseudocode)
const result = await ctx.subagents.start({
  prompt: [{ type: 'text', text: 'Analyze this code and find all bugs.' }],
  parent: currentAgent,
  signal,
  outputSchema: {
    type: 'object',
    properties: {
      bugs: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            file:        { type: 'string' },
            line:        { type: 'number' },
            severity:    { type: 'string', enum: ['critical', 'major', 'minor'] },
            description: { type: 'string' },
          },
        },
      },
    },
  },
})

// result.text is the subagent's text output (if any)
// result.structured is a type-safe JSON object — use it directly, no parsing
console.log(result.structured.bugs)
// → [{ file: 'auth.ts', line: 42, severity: 'critical', description: '...' }, ...]

This keeps data flow clean across the multi-Agent system: the subagent's analysis becomes the parent Agent's decision input directly, with no extra parsing layer in between.

Six Subagent Providers

dsh supports multiple underlying implementations for different deployment scenarios:

Provider	Description	Best For
`spawn-in-process`	Creates a new Agent instance in the same process	Local development, testing
`fork-in-process`	Forks the current Session; subagent inherits the prefix history	Subtasks that need to inherit context
`dsh-sdk`	Launches an isolated runtime via the dsh SDK	Subagents that need full isolation
`acp`	Communicates with remote Agents via the ACP protocol	Remote Agent clusters
`codex`	Calls a Codex Agent	Code-specific tasks
`claude-code`	Calls Claude Code	Code-specific tasks

For most local development, spawn-in-process is sufficient. When you need isolation — say, different subagents with different filesystem permissions — consider dsh-sdk.

Experimental: Agent Teams

Subagents follow a hierarchical model — parent-child relationships are explicit, communication paths are strict. But some collaboration patterns are inherently 'flat': multiple specialist Agents working on the same task in parallel, each reporting results as they finish.

Agent Teams is dsh's work-in-progress multi-Agent collaboration framework, accessible via ctx.agentTeams (experimental service):

roster: register team members, each with a name and role
mailbox: members send and receive messages via their mailbox
task board: a shared task list the team can claim and submit work against
coordinator mode: a coordinator Agent can distribute tasks and collect results

Compared to the Subagent hierarchy, Teams is closer to actual teamwork: no strict parent-child structure, members can communicate as peers, and tasks can be assigned dynamically.

Subagent model (hierarchical):       Agent Teams model (peer-to-peer):

     Orchestrator                        ┌──────────────────┐
    /     |      \                       │    task board     │
  Sub A  Sub B  Sub C                   └──────────────────┘
(strict parent-child,               Agent A ◄──► Agent B
 one-way delegation)                    │              │
                                        └────► Agent C ◄┘
                                        (peer messaging, dynamic collaboration)

Note: Agent Teams is currently experimental. The API may change in future releases. Confirm version stability before using in production.

Hands-On: Parent Agent Calling a Subagent

Here's a complete pattern: a parent Agent defines a tool that internally launches a subagent to perform a specialized analysis task.

// Pseudocode: parent Agent tool that delegates to a subagent for code analysis
const analyzeCodebaseTool = defineTool({
  name: 'analyze_codebase',
  description: 'Spawn a specialized subagent to analyze a given directory.',
  parameters: {
    directory:   { type: 'string', required: true, description: 'Directory to analyze' },
    focus:       { type: 'string', required: true, description: 'Analysis focus (e.g., naming conventions, circular dependencies)' },
  },

  async execute(args, exec) {
    // Launch the subagent (pseudocode — actual call is ctx.subagents.start())
    const result = await ctx.subagents.start({
      prompt: [{
        type: 'text',
        text: `Analyze the directory ${args.directory}. Focus on: ${args.focus}. Check each file and produce a structured issue list.`,
      }],
      parent: exec.agent,   // pass the parent Agent to establish lineage
      signal: exec.signal,  // pass cancellation signal — if parent is cancelled, subagent stops too

      // Subagent can only read files, not modify code
      toolFilter: {
        allow: ['read_file', 'list_files', 'search_files'],
      },

      // Specialized persona: code review expert
      persona: 'You are a code review expert. Read code carefully, identify issues, and provide a structured summary. Do not modify any files.',

      // Require structured JSON output
      outputSchema: {
        type: 'object',
        properties: {
          issues: {
            type: 'array',
            items: {
              type: 'object',
              properties: {
                file:     { type: 'string' },
                issue:    { type: 'string' },
                severity: { type: 'string', enum: ['high', 'medium', 'low'] },
              },
            },
          },
          summary: { type: 'string' },
        },
      },
    })

    // result.structured is type-safe JSON — use it directly
    const { issues, summary } = result.structured
    return `Analysis complete. Found ${issues.length} issue(s).\n${summary}`
  },
})

The core value of this pattern: the parent Agent doesn't need to know how the subagent analyzes code — it just receives structured results and makes its next decision.

Comparison with Other Frameworks

Framework	Multi-Agent Model	Key Difference from dsh
Claude Code Agent tool	Also spawns subagents	Simpler interface, but no `toolFilter` / `persona` / `outputSchema`
LangGraph	Graph-based workflow, fixed topology	dsh Subagents are dynamic — the model decides when to delegate at runtime, not via a predefined graph
AutoGen	Conversational multi-Agent, Agents message each other directly	dsh is hierarchical with explicit parent-child relationships; sibling Agents cannot communicate directly

In short: LangGraph suits scenarios with fixed, predictable flows; dsh Subagents suit scenarios where the flow is determined dynamically by the model.

What's Next in the Series

The next article covers observability: how do you know what an Agent did while it was running? What mechanisms does dsh provide for tracing tool calls, Token usage, and subagent lineage trees — and how do you hook into a logging system in production?

Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.

Find more on my Homepage