DEV Community: WonderLab

AI Evaluation Series (03): LLM-as-Judge — How to Use LLMs to Evaluate LLMs Correctly

WonderLab — Tue, 21 Jul 2026 03:44:16 +0000

How LLM-as-Judge Works

LLM-as-Judge delegates the quality evaluation of one LLM's output to another LLM (or the same one).

Two basic forms:

Pointwise scoring (single answer)

JUDGE_PROMPT = """Evaluate this AI response (1-10 per dimension):
Dimensions: accuracy, relevance, clarity
Question: {question}
Response: {answer}
Return JSON: {{"accuracy": int, "relevance": int, "clarity": int}}"""

Pairwise comparison (two answers)

COMPARE_PROMPT = """Which answer is better?
Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}
Reply with only "A" or "B"."""

Pairwise works well for comparing two versions (Prompt A vs Prompt B). Pointwise works better for monitoring a single system's quality trend over time.

Three Experiments: How Severe Are the Biases?

Experiment 1: Position Bias

Design: Same answer pair, evaluated twice — first with A shown first, then with B shown first. Without bias, the winner should be independent of order; first-position win rate should be ~50%.

def experiment_position_bias():
    for pair in answer_pairs:
        winner_order1 = judge_pairwise(question, answer_a, answer_b)  # A first
        winner_order2 = judge_pairwise(question, answer_b, answer_a)  # B first
        # Unbiased: winner_order1 == winner_order2 rate should approach 100%

Results:

First-position win rate:  67%  (unbiased baseline: 50%)
Second-position win rate: 33%
Consistency rate:         67%  (unbiased baseline: 100%)

Position bias magnitude:  17% above 50/50 baseline

A consistency rate of 67% means 33% of judgments flip when the same answer pair swaps order. If you use this Judge for A/B testing, one third of the results track presentation order, not answer quality.

Experiment 2: Verbosity Bias

Design: Same question, two answer versions — a concise version (2-3 sentences) and a verbose version with identical core content but padded with filler, repetition, and elaboration. Content is equivalent; only length differs.

Concise version:
Lists are mutable (add/remove/change); tuples are immutable.
Tuples are faster, use less memory, and work as dictionary keys or multi-return values.

Verbose version:
That's a great question! Let me explain in detail...
(same core information, expanded to ~300 words with intros, repetition, and summary)

Results:

Avg concise answer score:  7.5/10
Avg verbose answer score:  8.0/10
Verbosity bonus:           +0.5 points
(unbiased baseline: 0.0 point difference)

Equal content, and the longer version scores 0.5 points higher. Evaluating two prompts where one produces concise answers and the other verbose ones, the Judge systematically favors verbosity — independent of which version actually helps the user more.

Experiment 3: Framing Bias

Design: Same answer, evaluated with two prompts — a standard Judge prompt vs a strict expert prompt (explicitly stating that ordinary answers don't exceed 6/10, only excellent ones reach 8+).

Results:

Standard prompt avg score:      8.0/10
Strict expert prompt avg score: 8.0/10
Framing delta:                  +0.00 points

What this means: glm-4-flash doesn't respond to the strictness level specified in the prompt — even with an explicit higher standard, scores stay the same. This is useful information: for this model, you don't need to worry that prompt wording will systematically inflate or deflate scores. But stronger models (GPT-4, Claude) may show more framing sensitivity. Always test before assuming.

Three Mitigation Strategies

Mitigate Position Bias: Multi-Round Randomization

Single evaluations are unreliable with 33% flip rate. Fix: evaluate each pair multiple times with randomized order.

import random

def pairwise_with_randomization(judge, question, answer_a, answer_b, rounds=5):
    a_wins = 0
    b_wins = 0
    for _ in range(rounds):
        if random.random() > 0.5:
            result = judge(question, answer_a, answer_b)
            if result == "A": a_wins += 1
            else: b_wins += 1
        else:
            result = judge(question, answer_b, answer_a)
            if result == "A": b_wins += 1  # "A" position holds B
            else: a_wins += 1

    return {"a_win_rate": a_wins / rounds, "b_win_rate": b_wins / rounds}

5 rounds of randomization averages position bias toward 50%. Cost is 5x API calls; reliability is substantially higher.

Mitigate Verbosity Bias: Per-Dimension Scoring + Anti-Verbose Instructions

Option A: Score dimensions separately, no combined score

ANTI_VERBOSE_JUDGE = """Evaluate this response. Score each dimension 1-5 independently:

1. Accuracy: Is the technical content correct? (length-independent)
2. Conciseness: Is the information density high? (more concise = higher score)
3. Relevance: Does it directly answer the question?

Note: longer answers are not better — evaluate content quality only."""

Option B: Explicit declaration in the prompt

"Note: response length does not affect scoring. A concise accurate answer and a detailed accurate answer should receive equal scores."

Mitigate Self-Bias: Use a Stronger or Different Judge Model

If you generate with one model and evaluate with the same model, self-bias leads to systematic overestimation.

# Generate with cheaper model
generator = ChatOpenAI(model="glm-4-flash", ...)

# Evaluate with stronger or different-vendor model
judge = ChatOpenAI(model="gpt-4o", ...)
# or
judge = ChatAnthropic(model="claude-sonnet-4-6", ...)

Cost increases; evaluation reliability improves. For important release decisions, use Claude to evaluate GPT output (or vice versa) rather than self-evaluation.

Production-Ready Judge Prompt Template

Incorporating all three mitigations:

PRODUCTION_JUDGE_PROMPT = """You are a strict technical content reviewer.

Scoring rules:
- Score range 1-5, NOT 1-10 (reduces variance)
- Length does not affect scores — evaluate content quality only
- Vague or verbose writing is NOT a positive — clarity and conciseness are
- 3 means "meets expectations", not "barely acceptable"

Dimensions (each 1-5):
1. Accuracy: Is the technical content correct? Any factual errors?
2. Relevance: Does it directly answer the question? Any off-topic content?
3. Usefulness: Can a user act on this answer to solve their problem?

Question: {question}
Response: {answer}

Return JSON only:
{{"accuracy": int, "relevance": int, "usefulness": int, "reasoning": "one sentence explaining the lowest-scoring dimension"}}"""

Design choices:

1-5 not 1-10: reduces variance; LLMs are inconsistent at fine-grained distinctions (7 vs 8)
"3 = meets expectations": prevents LLMs from treating 3 as negative, which causes score inflation across the board
Require reasoning: forces the Judge to explain the lowest-scoring dimension, reducing arbitrary scoring
Anti-verbosity declaration: explicitly counters verbosity bias

Practical Recommendations

Single evaluations have limited reliability: With 33% flip probability from position bias, single results aren't enough to make decisions. For important comparisons, run at least 3 rounds and report win rates, not single outcomes.

Calibrate before scaling: Before running a new Judge Prompt at scale, test it against 20-30 human-annotated samples. Spearman correlation between Judge scores and human ratings should exceed 0.7 before using the Judge for decision-making.

Monitor score distributions: If the Judge assigns 90% of outputs a score of 7-8, the distribution is too concentrated and the Judge lacks discriminative power. Adjust the prompt or use a stronger model.

Use scenario-specific Judges: A code generation Judge should focus on correctness and runnability; a summarization Judge should focus on faithfulness; a conversational AI Judge should focus on helpfulness. A generic Judge applied to all scenarios loses evaluation precision.

Summary

Three experiments produce three specific action items:

Position bias (67% first-position win rate): Pairwise evaluation must randomize order across multiple rounds — 33% flip probability makes single-shot results unreliable
Verbosity bias (+0.5 points): Declare "length doesn't affect scoring" in the prompt, and add a conciseness dimension to counteract
Framing bias (0.0 delta on glm-4-flash): This model doesn't respond to prompt strictness level — but different models behave differently, always verify before assuming

References

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
Full demo code: eval-03-llm-judge

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project of the Day (#128): BISHENG — Open-Source Enterprise LLM DevOps Platform

WonderLab — Tue, 21 Jul 2026 03:42:59 +0000

Introduction

"Bi Sheng invented movable type printing. BISHENG helps enterprises 'typeset' large language model capabilities into their own business processes with the same kind of flexible, recomposable modularity."

This is article #128 in the Open Source Project of the Day series. Today's project is BISHENG — DataElem's open-source enterprise LLM application DevOps platform.

Enterprises adopting LLMs frequently encounter a painful fragmentation: one tool specializes in RAG, another in workflow orchestration, another in model management, and yet another in permissions. Each solves one piece. Stitching them together creates a maintenance burden that compounds over time.

BISHENG positions as a one-stop solution: document parsing, knowledge base construction, RAG retrieval, visual workflow orchestration, multi-agent collaboration, model fine-tuning management, and enterprise permissions governance — all in a single platform.

What You'll Learn

AGL (Agent Guidance Language) framework: how to encode domain expert experience into AI agents
Visual workflow builder's Human-in-the-Loop design
High-precision document parsing: OCR trained on 5+ years of proprietary data
Enterprise RAG use cases: from policy comparison to resume screening
BISHENG's technical architecture: Milvus + Elasticsearch + OnlyOffice + LLaMA-Factory
v2.6.0's Linsight Task Mode

Prerequisites

Familiarity with enterprise LLM application concepts (RAG, Agent, workflow)
Experience with or interest in enterprise AI system development
Docker basics (for local deployment)

Project Background

Why "Bi Sheng"

The project is named after Bi Sheng, the 11th-century inventor of movable type printing. The metaphor is fitting: where woodblock printing was rigid and inflexible, movable type was modular and recomposable. BISHENG aims to bring the same flexibility to enterprise AI capability building — not hardcoded fixed pipelines, but modular, orchestratable, iterable components.

Author / Team

Company: DataElem (Shanghai)
License: Apache-2.0
Version: v2.6.0 (July 2026)
User base: Fortune 500 companies and industry-leading organizations

Project Stats

⭐ GitHub Stars: 11,600+
🍴 Forks: 1,900+
📦 Releases: 73
📄 License: Apache-2.0
💻 Language: Python ~50% + TypeScript ~47.5%

Core Features

Linsight Agent (AGL Framework)

AGL (Agent Guidance Language) is BISHENG's proprietary agent guidance framework, addressing a core enterprise AI problem: how do you encode a specific domain expert's experience, preferences, and business logic into an AI, so it operates at expert level in professional scenarios?

A generic LLM performing financial analysis might produce technically correct but business-noncompliant conclusions. AGL lets you explicitly embed compliance specialists' judgment logic, common business edge cases, and industry-specific analytical perspectives into the agent's decision process.

Linsight Task Mode (v2.6.0): Agents don't just answer questions — they can proactively plan and execute multi-step tasks, delivering formatted reports that meet enterprise requirements.

Visual Workflow Builder

This is BISHENG's most differentiating core capability:

Single framework for all task types. Many platforms separate "chatflow" and "workflow" into independent modules with incompatible interfaces. BISHENG uses one visual workflow engine for every scenario.

Supported flow control:

Loop: Agent repeats until a condition is met
Parallelism: Multiple branches execute simultaneously
Batch processing: Execute the same flow for each item in a list
Conditional branching: Route to different paths based on intermediate results

Human-in-the-Loop:

An essential capability for enterprise scenarios:

Workflow running...
    ↓
Reaches "Contract Risk Assessment" node
    ↓
Legal Agent outputs preliminary assessment
    ↓
[PAUSE — waiting for human review]
    ↓  Human reviews, approves / modifies / rejects
    ↓
Continue with subsequent steps

This design inserts human judgment at high-risk decision points, rather than running fully automated from end to end. For enterprises with strict compliance requirements (finance, legal, healthcare), this is a hard requirement, not a nice-to-have.

High-Precision Document Parsing

BISHENG has a distinctive advantage in document parsing — not an integrated third-party tool, but a proprietary model DataElem trained on 5+ years of proprietary data:

Coverage:

Printed text recognition (common and rare characters)
Handwritten text recognition (including Chinese handwriting)
Complex table recognition (including merged cells across rows and columns)
Document layout analysis (multi-column, embedded images, header/footer separation)
Seal/stamp recognition
Chart and figure recognition

Deployment: Fully private deployment, no external API calls required, data never leaves the intranet.

This capability is particularly valuable in Chinese enterprise document processing: vast amounts of scanned paper forms, handwritten approval slips, and stamped documents are persistent pain points for traditional OCR tools. BISHENG's models have domain-specific optimization for exactly this type of data.

Enterprise RAG Knowledge Base

Complete enterprise knowledge base pipeline:

Source documents (PDF/Word/Excel/scans)
        ↓
Document parsing (high-precision OCR + layout analysis)
        ↓
Text chunking + vectorization (Milvus)
        ↓
Hybrid retrieval (vector similarity + full-text search via Elasticsearch)
        ↓
Knowledge base Q&A / citation tracing

Typical enterprise scenarios:

Scenario	Description
Policy comparison	Clause-by-clause diff of new vs. old policy versions, generating change reports
Contract review	Automated comparison of contract terms against company standards, flagging risk points
Meeting minutes	Audio/video → structured minutes, automatic action item tracking
Resume screening	Multi-dimensional matching of candidate resumes against job requirements, scoring
Call record analysis	Structured analysis of customer service calls, key information extraction
Unstructured data governance	Historical document classification, tagging, knowledge graph construction

Unified Model Management

Full lifecycle management from model selection to production operation:

Model integration: OpenAI-compatible API / local deployment (llama.cpp, vLLM, etc.)
Dataset management: Version control and quality control for labeled data
SFT fine-tuning: LLaMA-Factory integration for fine-tuning on private data
Evaluation: Build evaluation sets, quantify model performance on business scenarios
Deployment management: Different departments using different model versions, traffic control, A/B testing

Enterprise-Grade Features

Permissions and security:

RBAC (role-based access control)
User group management (different permissions per department)
SSO single sign-on (LDAP support)
Per-group traffic control quotas

Operations and compliance:

High-availability deployment
Vulnerability scanning and security patching
Usage statistics and audit logs
Observability monitoring

Document editing:

OnlyOffice integration for online collaborative editing of Word/Excel/PPT
Fits the "AI generates draft → human edits online → final output" workflow pattern

Quick Deployment

System requirements:

CPU: ≥ 4 cores (18 vCPUs recommended)
RAM: ≥ 16 GB (48 GB recommended)
Docker 19.03.9+, Docker Compose 1.25.1+

git clone https://github.com/dataelement/bisheng.git
cd bisheng/docker
docker compose -f docker-compose.yml -p bisheng up -d

Access via http://IP:3001. First registered user automatically becomes system admin.

Technical Architecture

Frontend (React + TypeScript)
    ↓
Backend (Python + FastAPI)
    ├── Workflow engine (LangChain/Langflow-based, customized)
    ├── Document parsing (proprietary OCR models)
    ├── Agent framework (AGL framework)
    └── Model management (LLaMA-Factory integration)
    ↓
Data layer
    ├── Milvus (vector storage)
    ├── Elasticsearch (full-text search)
    ├── MySQL (metadata)
    └── MinIO/S3 (file storage)

Acknowledgements: Built on LangChain, Langflow, Unstructured, and LLaMA-Factory, with extensive enterprise customization on top.

Links and Resources

🌟 GitHub: dataelement/bisheng
🌐 Website: bisheng.dataelem.com
📖 Documentation: github.com/dataelement/bisheng/wiki

Conclusion

BISHENG's core value is reducing the integration complexity of enterprise AI development.

Building an LLM application for an enterprise typically requires separately selecting a document parsing tool, RAG framework, workflow orchestration platform, agent framework, model management system, and permissions infrastructure — then connecting them all together. That integration process alone consumes substantial engineering resources. BISHENG consolidates all of these into a single platform with a unified data model, unified permissions, and unified monitoring.

High-precision document parsing is an underappreciated differentiator. In Chinese enterprise contexts, vast amounts of critical information still exists in scanned documents, handwritten forms, and stamped records. The quality of processing that data directly determines knowledge base usability. DataElem's 5+ years of proprietary training data represents a real advantage on this dimension.

The AGL framework and Human-in-the-Loop workflow reflect serious consideration of what makes enterprise scenarios different from general-purpose AI: not all decisions can be fully automated, and the right design preserves space for human intervention at critical junctures.

11.6k Stars and production use by Fortune 500 companies indicate this is not just a technical proof of concept — it's a product delivering measurable value.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

AI Evaluation Series (02): Metric Design — From Business Goals to Measurable Indicators

WonderLab — Sun, 19 Jul 2026 01:11:20 +0000

What Happens Without Metrics

A RAG Q&A system launches. The engineers say "all tests passed" — API response time under 2 seconds, correct format, no crashes.

Two weeks later, users report "the AI often doesn't answer the actual question." Investigation reveals: retrieval recall is only 40%. More than half the user questions can't find relevant documents.

The problem was there before launch. No one had designed a "retrieval quality" metric, so no one saw it.

The L1/L2/L3 Framework

Metrics organize into three layers, each addressing a different class of question:

L1 — Business Outcome
  The end goal: is the AI system creating value?
  Examples: task completion rate, user adoption rate, user satisfaction

L2 — Output Quality
  The middle layer: how good is the AI's actual output?
  Examples: accuracy, relevance, completeness

L3 — System Health
  The foundation: is the system running stably?
  Examples: response latency, token cost, failure rate

Layer dependencies:

L3 fails → L2 degrades (timeouts cause truncated output) → L1 drops (tasks fail)
L3 healthy, L2 poor → L1 still low (users don't adopt low-quality output)
All three healthy → L1 reflects real value

Start diagnosing from L3 upward; reverse from L1 downward is much slower.

Metric Selection by Scenario

Scenario 1: Document Q&A (RAG)

L3 System health:
  Response latency P90     → < 5s
  Token cost per query     → < 3000
  Retrieval failure rate   → < 1%

L2 Output quality:
  Faithfulness             → no hallucinations beyond retrieved content (RAGAS)
  Answer Relevancy         → actually addresses the question (RAGAS)
  Context Precision        → fraction of retrieved content that's useful (RAGAS)
  Context Recall           → fraction of relevant content that was retrieved (RAGAS)

L1 Business outcome:
  Task completion rate     → fraction of users who mark "question resolved"
  Adoption rate            → fraction of users who copy/cite the AI response

The critical metric: Context Recall — the most commonly skipped and most important RAG metric. If retrieval doesn't find the relevant content, generation quality is irrelevant.

Scenario 2: Code Generation

L3:
  Response latency P90   → < 10s (code generation is slower; allow more)
  Excessive output rate  → avoid generating far more code than needed

L2:
  Syntax correctness     → can the output be parsed? (automatable)
  Test pass rate         → does the output pass unit tests? (automatable)
  Usability              → how much editing is needed? (human evaluation)
  Security scan pass     → does the output contain known vulnerabilities? (automatable)

L1:
  Adoption rate          → fraction of suggestions the user accepts
  Edit distance          → how much the user changed the suggestion (less = better)

The critical metric: Test pass rate — fully automatable, measurable per commit, a natural fit for CI integration.

Scenario 3: Document Summarization

L2:
  Faithfulness         → summary adds nothing not in the source (most important)
  Coverage             → source's key points appear in the summary
  Conciseness          → summary is noticeably shorter than source (otherwise no value)
  Readability          → LLM-as-Judge evaluation

The critical metric: Faithfulness. Summarization's greatest risk is "adding" information that wasn't in the source (hallucination) — more harmful than omitting a detail.

Scenario 4: Agent Task Completion

L2:
  Tool call accuracy   → correct tool selected, correct arguments (automatable)
  Step efficiency      → how many steps to complete the task (fewer is better)
  Trajectory quality   → is the reasoning path sound? (LLM-as-Judge)

L1:
  Task completion rate → did the task get done? (the core metric)
  First-run resolution → fraction completed without follow-up questions

The critical metric: Task completion rate, but define "completed" carefully — "artifact exists" versus "artifact meets quality standard" are different requirements.

Three Traps

Trap 1: Measuring Only L3

✗ Wrong:
  "API returned 200, latency 1.2s, this version can go to production."

Problem: L3 health doesn't mean business value. The RAG example above — L3
         completely normal, Context Recall at 40%.

Every release needs at least one L2 sampling evaluation.

Trap 2: Using BLEU/ROUGE for Semantic Quality

# Looks reasonable
rouge_score = rouge.compute(predictions=[output], references=[reference])

# The problem:
reference = "The capital of France is Paris."
output_1  = "Paris is the capital city of France."  # same meaning, low ROUGE
output_2  = "The capital of France is Paris, the capital."  # repetition, high ROUGE

# ROUGE scores output_2 higher, but output_2 is clearly worse

BLEU/ROUGE measure word overlap, not semantic correctness. For generative outputs, use LLM-as-Judge instead. BLEU/ROUGE only work for tasks with standard reference answers, like machine translation.

Trap 3: Setting Thresholds by Instinct

✗ Wrong:
  "Let's set Faithfulness > 0.8 as the quality gate."
  (Why 0.8? It sounds reasonable.)

✓ Right:
  Step 1: Run 100 samples, get the current baseline (e.g., mean = 0.73)
  Step 2: Manually inspect samples with Faithfulness < 0.6 — are they acceptable?
  Step 3: Set threshold based on that inspection (e.g., 0.65)
  Step 4: Document why samples below that threshold are unacceptable

Thresholds come from data distributions and business acceptability, not from alignment with round numbers.

Complete Metric Spec: Enterprise Document Q&A System

# eval_metrics.yaml
system: document-qa
version: "1.0"

metrics:
  l3_system_health:
    - name: response_latency_p90
      target: "< 5000ms"
      collection: trace_log
      alert_threshold: 8000ms

    - name: token_cost_per_query
      target: "< 2000 tokens"
      collection: llm_callback
      alert_threshold: 4000

    - name: retrieval_error_rate
      target: "< 1%"
      collection: error_log

  l2_output_quality:
    - name: faithfulness
      tool: ragas
      target: "> 0.80"
      collection: weekly_sampling (n=100)
      alert_threshold: 0.70

    - name: answer_relevancy
      tool: ragas
      target: "> 0.75"
      collection: weekly_sampling

    - name: context_recall
      tool: ragas
      target: "> 0.70"
      collection: weekly_sampling
      note: "Most important retrieval quality metric"

    - name: format_compliance
      tool: rule_check
      target: "100%"
      collection: every_request

  l1_business_outcome:
    - name: task_completion_rate
      target: "> 70%"
      collection: user_feedback_widget
      note: "Fraction of users clicking 'this solved my question'"

    - name: adoption_rate
      target: "> 50%"
      collection: behavior_tracking
      note: "Fraction of users who copy/cite the AI response"

Roadmap to a Working Metric System

Step 1 (no tools needed, do now):
  □ Write down the system's core business goal (what is L1?)
  □ List 3-5 most important L2 output quality dimensions
  □ Confirm L3 basics are monitored (latency, error rate)

Step 2 (within 1 week):
  □ Manually evaluate 50 real use cases to get baseline numbers per L2 metric
  □ Set thresholds based on that baseline — not on intuition

Step 3 (ongoing):
  □ Run L2 sampling eval before each release; compare to baseline
  □ Check L1 data monthly; verify it moves consistently with L2

Summary

L1/L2/L3 each have a distinct role: L3 monitors system stability, L2 evaluates output quality, L1 validates business value — missing any one layer creates a blind spot
Scenario determines critical metrics: RAG systems need Context Recall above all; code generation needs test pass rate; Agents need task completion rate — copying another system's metrics doesn't mean they fit yours
Thresholds come from data: run a baseline first, then set the threshold — don't set a threshold and then justify it with data afterward

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project #127: wigolo — Zero API Key, $0/Query Local-First Web Intelligence for AI Agents, Compared Against Tavily / Exa / Firecrawl

WonderLab — Sun, 19 Jul 2026 01:09:31 +0000

Introduction

"The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query."

This is article #127 in the "One Open Source Project a Day" series. Today's project is wigolo — a local-first web intelligence tool built specifically for AI agents, delivering search, crawl, fetch, and research capabilities over MCP with no API keys and zero per-query cost.

Equipping an AI agent with "web search" is a problem that looks simple on the surface but has surprisingly deep layers. Tavily, Exa, and Firecrawl have each carved out mature niches in the cloud space — but they share a fundamental trait: per-query billing, data leaving your machine, and mandatory account registration. wigolo's premise: search engines' raw data is public, ranking and extraction algorithms can run locally, and the agent's query history can be cached into a local semantic index. Why should every agent search cost money and travel to the cloud?

1,218 Stars. Created April 2026. Public beta.

What You'll Learn

wigolo's 10 tools and the design logic behind each
What "byte-pinned source provenance" and "explainable scores" mean, and why they matter
Local-first architecture: what runs on-device, what optionally reaches the cloud
Head-to-head comparison with Tavily, Exa, Firecrawl — each tool's differentiated positioning
What scenarios favor wigolo, and where cloud tools are still the better choice

Prerequisites

Experience with Claude Code or any AI agent
Basic familiarity with MCP (Model Context Protocol)
General understanding of AI agent tool-calling mechanics

Project Background

Overview

wigolo is a Node.js process that exposes 10 web-related tools as an MCP server, letting AI agents call search, fetch, crawl, extract, and other functions over the standard MCP protocol.

Core design principles:

Local-first: cache, vector index, embedding model, and reranker all run locally (~/.wigolo/)
No forced API keys: the six core tools — search, fetch, crawl, extract, cache, find_similar — require zero API keys
Explicit failure: bot blocks, dead engines, and stale cache hits are reported in results rather than silently dropped or disguised as empty results

Project Info

Author: KnockOutEZ
Language: TypeScript
License: AGPL-3.0
Status: Public Beta
Website: knockoutez.github.io/wigolo

Project Stats

⭐ GitHub Stars: 1,218+
🍴 Forks: 83+
📄 License: AGPL-3.0
📅 Created: 2026-04-12

Quick Setup

# One command: download browser engine and local models, write MCP config
npx wigolo init --agents=claude-code

# Configure multiple agents at once
npx wigolo init --agents=claude-code,cursor,codex

# Check all component health
npx wigolo doctor

Requires Node ≥ 20 and about 1.5 GB of disk space (browser engine + local ML models).

All six core tools work immediately after setup — no API keys needed. If you want research and agent tools to produce a synthesized answer (rather than raw data for the host LLM to process), a free Gemini key is the single biggest quality upgrade:

export WIGOLO_LLM_PROVIDER=gemini
export GEMINI_API_KEY=<free key from aistudio.google.com>

For fully air-gapped operation, use a local Ollama model instead: WIGOLO_LLM_PROVIDER=ollama.

10 Tools, Explained

Core Retrieval Tools

search — Multi-engine parallel search

18 direct engine adapters (no intermediate proxy API)
Rank fusion across engines + ML reranking
Pass an array of queries to fan out multiple searches in a single MCP call
Each result includes an explainable score: semantic + lexical + engine_consensus broken out separately

fetch — Tiered-routing page retrieval

Plain HTTP request
    ↓ (fails or detects SPA signals)
Headless browser (handles JS-rendered content)
    ↓ (encounters bot challenge)
Challenge clearance flow
    ↓ (clearance fails)
Flags blocked_by_challenge — does NOT pretend it succeeded

wigolo learns which tier each domain requires and skips unnecessary escalations on repeat visits.

crawl — Multi-page crawling

Supports BFS, DFS, sitemap mode, or map-only (just build the site map, don't download content). Respects robots.txt, enforces per-domain rate limits, and deduplicates boilerplate templates.

extract — Structured extraction

Extracts from a page: tables, JSON-LD, metadata, brand assets, named schemas (Article / Recipe / Product / …), or any arbitrary structure via a custom JSON Schema.

Memory and Discovery Tools

cache — Local semantic cache

Every visited page is automatically cached. Supports both keyword and semantic similarity queries, works offline. Also exposes cache stats, clear, and change detection.

find_similar — Similar content discovery

Find pages similar to a given URL or concept, via three-way fusion: keyword + semantic + live web.

Research and Autonomous Tools

research — Deep research

Decomposes a question → fires parallel sub-queries → fetches sources → synthesizes a cited report (or generates a structured summary for the host LLM to write).

agent — Autonomous gather loop

Self-plans → searches → fetches → extracts → synthesizes, with configurable time budget and optional output JSON Schema.

Monitoring Tools

diff + watch — Page change detection

Check what changed on a URL since the last visit; set up recurring monitoring and push changes to a webhook.

Deep Dive: Two Unique Features

Byte-Pinned Source Provenance

This is wigolo's most distinctive technical feature — one that no other tool in the comparison set offers. Each search result carries not just a snippet, but:

{
  "excerpt": "Logical replication is a method of replicating data objects…",
  "citation_id": "src-1",
  "source_span": { "start": 1042, "end": 1305 },
  "evidence_score": {
    "final": 0.86,
    "semantic": 0.91,
    "lexical": 0.78,
    "engine_consensus": 3
  }
}

source_span is a byte-offset range pinning the excerpt to its exact location in the original document — not "somewhere in this article," but a precise, verifiable position.

evidence_score breaks down into three independently interpretable components: semantic relevance, lexical match, and engine consensus (how many engines returned this result). Weak results are flagged as junk by wigolo's own scorer and surfaced in the output rather than silently filtered.

Why it matters: When an AI agent builds a RAG pipeline or answers questions that need citations, byte-level provenance means references are actually verifiable rather than plausibly attributed. The "AI might be fabricating citation locations" uncertainty drops by one layer.

Local-First Architecture

AI agent (any MCP client / REST / SDK)
    ↓
wigolo process (single Node process)
    │
    ├── Search layer
    │   18 direct engine adapters
    │   rank fusion + ML reranking (local model)
    │
    ├── Fetch layer
    │   Tiered routing (HTTP → headless browser → challenge clearance)
    │   Per-domain learning + cache reuse
    │
    ├── Local storage (~/.wigolo/)
    │   Keyword index + vector index (local embedding model)
    │   Browser engine + ML models
    │
    └── Optional cloud (dashed)
        LLM API (only for research/agent synthesis)
        Aggregator backend (widen search funnel, legacy mode)

What runs locally:

✅ Direct search engine adapters
✅ ML reranking model
✅ Vector embedding model (all-MiniLM-L6-v2 etc.)
✅ Headless browser engine
✅ Local cache (SQLite + vector index)

What's optionally cloud:

🔲 LLM synthesis (when research/agent needs a synthesized answer)
🔲 Aggregator backend (widen the search funnel, optional)

Head-to-Head Comparison: wigolo vs Tavily / Exa / Firecrawl

This is the main event. wigolo's README includes a comparison table, but stops short of explaining each tool's actual positioning and optimal use case. Here's a deeper reading.

Feature Matrix

Capability	wigolo	Tavily	Exa	Firecrawl
Multi-engine web search	✅	✅	✅	✅
Page fetch + structured extract	✅	✅	✅	✅
Full-site crawl	✅	✅	—	✅
Byte-pinned source provenance	✅	—	—	—
Explainable score decomposition	✅	—	—	—
Persistent local cache (offline re-query)	✅	—	—	—
Query data stays on your machine	✅	—	—	—
API key / account required	None	Required	Required	Required
Per-query cost	$0	Metered	Metered	Metered

Each Tool's Actual Positioning

Tavily — Purpose-built search API for AI agents; currently the most widely integrated paid service in the "RAG + agent search" space. Strengths: production-grade quality out of the box, native support in LangChain/LlamaIndex and most major frameworks, search quality tuned specifically for agent use cases. Weakness: per-query billing, which can add up quickly in high-frequency agent workflows.

Exa — Positions itself as "semantic web search," designed to understand the intent behind a query rather than just matching keywords. Notable strength: high-quality retrieval of technical documentation and academic content — the benchmark section in wigolo's README notes that Exa successfully rendered the full comparison matrix from an official docs page that other tools missed. Like Tavily: requires registration + metered billing.

Firecrawl — Focused on web crawling and structured extraction, not search. Primary strength: handling complex JS-rendered pages, large-scale crawl jobs, and extracting structured data from arbitrary websites. The right tool for "given a set of URLs, batch-extract structured content" — not for "given a question, search for relevant content."

wigolo — Attempts to combine all three capabilities into a single local tool while eliminating API keys and per-query costs. The differentiated bets: byte-pinned provenance, explainable scores, local cache with offline re-query capability, and $0 core tooling cost.

Scenario Recommendations

Choose wigolo when:

Frequent technical documentation queries during development (no billing constraints)
Source traceability matters (byte-level pinning → agent citations are verifiable)
Privacy requirements prohibit query data leaving the machine
High agent query volume makes cloud API costs a concern
Building "memory-enabled" research tasks that benefit from local caching across sessions

Choose Tavily when:

Production-grade quality out of the box, no local dependencies to maintain
Already in the LangChain/LlamaIndex ecosystem needing framework-native integration
Query volume is manageable and per-query billing is acceptable

Choose Exa when:

Primary use case is technical documentation or academic content (semantic relevance priority)
Official documentation retrieval quality is critical (strong docs page rendering)

Choose Firecrawl when:

Batch crawling specific websites to extract structured content (not a search use case)
Large-scale web content processing pipelines

An Honest Limitation

wigolo's README flags one: data-center IP reputation scores lower than residential IPs on some sites' bot-detection systems, which can reduce challenge-clearance rates when running on a cloud server vs. a local machine. Self-hosted deployments should account for this; the README documents optional proxy configurations.

Other Integration Options

wigolo isn't just a Claude Code plugin. It supports multiple invocation paths:

REST API

wigolo serve   # starts on 127.0.0.1:3333

curl -sX POST http://127.0.0.1:3333/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"local-first software","max_results":5}'

Cross-loopback connections require a bearer token; fail-closed by design.

TypeScript / Python SDKs

from wigolo import local_client

with local_client() as client:
    res = client.search(query="local-first web search", max_results=5)
    for r in res["results"]:
        print(r["title"], r["url"])

The SDK automatically reuses a running daemon, or starts one if needed, and shuts it down as appropriate.

Framework Integrations

Framework	Package	What it exposes
LangChain	`wigolo-langchain`	Each tool as `BaseTool` + `BaseRetriever` for RAG
CrewAI	`wigolo-crewai`	`wigolo_tools()` passed directly to a crew
LlamaIndex	`wigolo-llamaindex`	`BaseReader` loading pages as Documents
Vercel AI SDK	`wigolo-vercel-ai-sdk`	Tool factory for `generateText`/`streamText`

Resources

Official Links

🌟 GitHub: KnockOutEZ/wigolo
🌐 Website: knockoutez.github.io/wigolo
📦 npm: wigolo
📄 Docs: docs/README.md

Summary

wigolo's core thesis: an AI agent's web access layer shouldn't be a perpetually-billed cloud API black box. Search engine raw data is public. Reranking and extraction algorithms can run locally. Query history can be cached into a local semantic index that persists across sessions. Put these together and you get a zero-cost, on-device, offline-capable web intelligence layer.

Two design decisions are worth singling out:

Byte-pinned source provenance: Unique among all the tools compared. When an agent cites a search result, source_span provides a byte-offset range that ties the excerpt to its exact position in the source document. For research tasks that require verifiable citations, this removes one layer of "the AI might be fabricating where this came from" uncertainty.

Explicit failure over silent pretense: Bot blocks surface as blocked_by_challenge. Dead engines report themselves. Weak results are scored and flagged by wigolo's own evaluator, then shown in the output rather than quietly discarded. This "always tell you what ground you're standing on" philosophy is significantly more reliable in agent contexts than tools that present silence or empty results when something goes wrong.

If your Claude Code workflow involves heavy documentation querying, technical information search, or if you're building an agent pipeline that needs web search — npx wigolo init --agents=claude-code is currently the lowest-cost entry point in the most literal sense: zero dollars, zero API keys, zero per-query billing.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

Open Source Project #126: code-review-graph — Build a Structural Map of Your Codebase to Cut AI Review Tokens by 82x

WonderLab — Sat, 18 Jul 2026 01:49:07 +0000

Introduction

"Stop burning tokens. Start reviewing smarter."

This is article #126 in the "One Open Source Project a Day" series. Today's project is code-review-graph — a local-first tool that builds a persistent structural map of your codebase and delivers precise context to AI coding tools via MCP.

AI coding agents have a hidden inefficiency during code reviews: they don't know "if I change this function, what else breaks?" — so they either read large portions of the codebase to build context, or rely on grep and hope for the best. code-review-graph's solution: pre-build a structural graph (functions, classes, call edges, inheritance, test coverage), then at review time query the graph to compute the "blast radius" of a change, and give the AI only the files that actually matter.

Benchmark: ~82x median token reduction. 528x best case.

19,762 Stars. Created February 2026. Supports 14 AI coding platforms.

What You'll Learn

How blast-radius analysis works: tracing from changed files to callers, dependents, and tests
The full architecture: Tree-sitter AST → SQLite graph → MCP tool chain
The incremental update mechanism: why a 2,900-file repo re-indexes in under 2 seconds
Three-tier edge confidence scoring (EXTRACTED/INFERRED/AMBIGUOUS) and what it means
An honest reading of the benchmark numbers: 528x is the best case, not the median

Prerequisites

Basic experience with Claude Code or other AI coding tools
Familiarity with MCP (Model Context Protocol)
Basic understanding of static analysis and call graphs

Project Background

Overview

code-review-graph (CRG) is a local-first code intelligence tool with four core functions:

Parse your codebase with Tree-sitter into a graph of functions, classes, imports, and call relationships
Persist the graph as a SQLite file in .code-review-graph/ — zero network dependencies
Expose 30 MCP tools so your AI assistant can query "what does this change affect?"
Support incremental updates — on a change, re-parse only the modified files

Author / Team

Author: tirth8205
Language: Python 3.10+
License: MIT
Version: v2.3.6
Website: code-review-graph.com

Project Stats

⭐ GitHub Stars: 19,762+
🍴 Forks: 2,107+
📄 License: MIT
📅 Created: February 26, 2026

Features

Quick Start

pip install code-review-graph        # or: pipx install code-review-graph
code-review-graph install            # auto-detects platforms, writes MCP configs
code-review-graph build              # parse codebase, build graph

Three commands. install detects every AI coding tool on your machine, writes the correct MCP configuration for each, injects graph-aware instructions into your platform rules files, and installs platform-native hooks/skills. Restart your editor, then ask the AI:

Build the code review graph for this project

Initial build: ~10 seconds for a 500-file project. After that, incremental.

Platform Support

One install, 14 platforms: Codex, Claude Code, CodeBuddy Code, Cursor, Windsurf, Zed, Continue, OpenCode, Antigravity, Gemini CLI, Qwen, Qoder, Kiro, GitHub Copilot.

Target-specific installs:

code-review-graph install --platform claude-code
code-review-graph install --platform cursor
code-review-graph install --platform codex

Token Savings Panel

┌─────────────────────── Token Savings ────────────────────────┐
│ Full context would be:     12,921 tokens                     │
│ Graph context used:           762 tokens                     │
│ Saved:                     12,159 tokens (~94%)              │
│ Breakdown: Functions 244 · Tests 191 · Risk 244 · Other 83   │
└──────────────────────────────────────────────────────────────┘

Printed by detect-changes --brief or update --brief. Add --verify to cross-check against OpenAI's cl100k_base tokenizer; the estimate stays within ~1% of real tokens in practice.

Deep Dive

Core Architecture

Repository
    │
    ▼ git ls-files (only tracked files indexed)
Tree-sitter parsers
    │ extracts: functions / classes / imports / calls / inheritance / test coverage
    ▼
SQLite graph database
(.code-review-graph/, git-ignored by default)
    │
    ├── MCP server (30 tools)
    │       ↓ AI assistant queries
    │   get_impact_radius_tool
    │   get_review_context_tool
    │   detect_changes_tool
    │   ...
    │
    └── CLI
        detect-changes --brief
        update --brief
        visualize
        ...

Graph nodes: functions, classes, files, modules
Graph edges: call relationships, import relationships, inheritance, test coverage

Blast-Radius Analysis

The core concept. When a file changes:

Changed file (e.g. login() in auth/login.py)
    ↓
Query graph: which functions directly call login()?
    ↓
Query graph: which functions call those callers?
            (configurable depth; default depth=2)
    ↓
Query graph: which test files cover these functions?
    ↓
Output: blast radius = minimal set of functions/files affected

The AI reads only this minimal set instead of scanning the whole codebase.

Accuracy note (from the README, unusually candid):

The current recall=1.0 is a graph-derived upper bound — the ground truth comes from the same graph edges the predictor walks, so the measurement is circular by construction
"Co-change mode" grades against files the author actually co-edited in the same commit — independent evidence from git history, not the graph — and will produce substantially lower numbers
Blast-radius analysis is deliberately conservative: it's better to flag extra files than to miss a broken dependency

Incremental Updates: < 2 Seconds

File saved (hook fires / watch mode / crg-daemon)
    │
    ▼
SHA-256 hash check: which files actually changed?
    │
    ▼
Re-parse only changed files
    │
    ▼
Find dependents of changed files, mark edges for refresh
    │
    ▼
Graph updated

Benchmark: 2,900-file repo → < 2 seconds
           500-file repo, initial build → ~10 seconds

Three-Tier Edge Confidence

Graph edges carry confidence levels:

Level	Meaning
EXTRACTED	Explicit call parsed directly from the AST — high confidence
INFERRED	Relationship derived via type inference or semantic analysis — medium confidence
AMBIGUOUS	Dynamic dispatch, polymorphism, or cases that can't be statically resolved — low confidence

This lets the AI filter by confidence when querying — avoiding false alerts from low-quality edges in dynamically-typed code.

30 MCP Tools

Organized by purpose:

Context retrieval (code review core)

get_minimal_context_tool — ultra-compact context, ~100 tokens, call this first
get_impact_radius_tool — blast radius of changed files
get_review_context_tool — token-optimized review context with structural summary
detect_changes_tool — risk-scored change impact analysis

Graph queries

query_graph_tool — query callers, callees, tests, imports, inheritance
traverse_graph_tool — free-form BFS/DFS from any node with token budget
semantic_search_nodes_tool — search code entities by name or meaning (requires optional embeddings)

Architecture analysis

get_architecture_overview_tool — auto-generated architecture map from community structure
get_hub_nodes_tool — most-connected nodes (architectural hotspots)
get_bridge_nodes_tool — chokepoints via betweenness centrality
get_surprising_connections_tool — unexpected cross-community coupling
get_knowledge_gaps_tool — isolated nodes, untested hotspots, structural weaknesses

Other

refactor_tool — rename preview, dead code detection
generate_wiki_tool — Markdown wiki from community structure
cross_repo_search_tool — search across all registered repos

In token-constrained environments, expose only what you need:

code-review-graph serve --tools query_graph_tool,detect_changes_tool,get_review_context_tool

Language Coverage

40+ languages and formats: Python, JavaScript/TypeScript/TSX, Go, Rust, Java, C/C++, C#, Ruby, Kotlin, Swift, PHP, Scala, Solidity, Dart, R, Elixir, Zig, Vue/Svelte SFCs, Jupyter/Databricks notebooks (.ipynb), Terraform, Ansible, and more.

For unsupported languages, drop a languages.toml in .code-review-graph/ — no fork or code changes:

[languages.erlang]
extensions = [".erl"]
grammar = "erlang"
function_node_types = ["function_clause"]
class_node_types = ["record_decl"]
import_node_types = ["import_attribute"]
call_node_types = ["call"]

CI Integration: GitHub Action

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: tirth8205/code-review-graph@v2.3.6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

Graph built and queried entirely on the CI runner. No source code sent to any external service. Posts a sticky risk-scored comment on each PR, updated in place on every push. Optional fail-on-risk input turns the review into a merge gate.

Multi-Repo Daemon

crg-daemon add ~/project-a --alias proj-a
crg-daemon add ~/project-b
crg-daemon start
crg-daemon status
crg-daemon stop

For editors that don't support hooks (Cursor, OpenCode, etc.). Watches multiple repos in the background, health-checks every 30 seconds, auto-restarts dead watchers. Config persists in ~/.code-review-graph/watch.toml.

Benchmark Numbers — An Honest Reading

The README handles the benchmark numbers with unusual candor:

Repo	Corpus tokens	Graph tokens	Reduction
fastapi	951,071	2,169	528x
code-review-graph	208,821	2,495	93x
gin	166,868	1,990	92x
flask	125,022	1,986	71x
express	135,955	3,465	41x
httpx	89,492	2,438	38x
Median			~82x

528x is the best case (fastapi, the largest corpus). The median is 82x.

The whole-corpus baseline is an upper bound no real agent pays: a competent agent would grep for identifiers and read only the best-matching files. The agent_baseline benchmark measures this more realistic baseline — pure-python grep, top-3 files by match count — and that comparison is the honest one.

For small single-file changes, graph context can exceed naive file reads (the structural metadata overhead is larger than the file content). The express results show this. The README calls it out rather than hiding it.

vs. Related Tools

Tool	Approach	CRG difference
LSP / language servers	Per-language, per-symbol precision	CRG: one persistent cross-language graph; LSP stays more precise per symbol
RAG / embeddings	Similarity chunks	CRG: structural edges from AST parsing; embeddings are optional and only assist search
grep / agentic search	One-hop lookups	CRG wins on multi-hop questions: impact radius, callers-of-callers, tests-for, affected flows

When NOT to use CRG: small repos, trivial single-file diffs, one-off questions. The overhead of maintaining the graph isn't worth it for these cases.

Resources

Official Links

🌟 GitHub: tirth8205/code-review-graph
🌐 Website: code-review-graph.com
📦 PyPI: code-review-graph
📄 Benchmark reproduction: docs/REPRODUCING.md
💬 Discord: discord.gg/3p58KXqGFN

Summary

code-review-graph addresses a problem that gets worse as codebases grow: an AI agent doesn't know that changing A affects B, so it either reads too many files or relies on grep. CRG pre-builds the "what affects what" map so agents can query it directly.

Three engineering decisions worth noting:

Local-first with zero telemetry: the graph lives in SQLite, cloud embeddings are opt-in (off by default), and the GitHub Action runs entirely on your CI runner. Source code stays on your machine.

Honest benchmarks: 528x is labeled "best case," the median 82x is the headline; recall=1.0 is explicitly called out as circular (graph-derived ground truth); small-commit overhead is acknowledged rather than hidden. This level of transparency is genuinely uncommon in developer tooling.

Incremental updates as the primary path: the initial build is a one-time cost. Two-second incremental updates mean hooks and watch mode can keep the graph current as you work, not just when you remember to rebuild.

For medium-to-large codebases (hundreds to thousands of files), frequent code review workflows, or teams that regularly ask "if I change this, what breaks?" — CRG's MCP integration path is currently one of the most direct available. Three commands to onboard, then the agent queries the graph automatically from there.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

AI Evaluation Series (01): Why AI Evaluation Is Hard — Uncertainty, Subjectivity, and Multiple Dimensions

WonderLab — Sat, 18 Jul 2026 01:35:10 +0000

What Traditional Software Testing Assumes

Traditional software testing rests on three assumptions:

Deterministic output: same input gives same output; tests are reproducible
Correctness is decidable: a function returns 42 — right or wrong, there's a clear standard
Single dimension: code either passes the test or it doesn't; no "kind of passes"

AI applications break all three.

Three Fundamental Problems

Problem 1: Non-Deterministic Output

The same Prompt gets different responses from Claude today versus tomorrow. Setting temperature=0 reduces randomness but doesn't eliminate it. Model updates silently change outputs for the same input.

Engineering implication:

# Traditional test
assert calculator(2, 3) == 5  # runs 100 times, passes 100 times

# AI evaluation
score1 = judge(llm.invoke("Explain the Transformer"))  # 4.2/5
score2 = judge(llm.invoke("Explain the Transformer"))  # 3.8/5
# 0.4 point gap — normal variance or a problem?

Sample multiple times and average. Three to five samples is usually enough; cost is 3-5x. Don't judge quality from a single run.

Problem 2: Subjective Quality

"Explain what a Transformer is" has countless correct answers. An explanation for a 10-year-old and one for an ML engineer can both be "good answers" while being completely different.

Subjectivity shows up as:

Same question, different good answers for different audiences
  → Depth: beginner vs expert
  → Style: analogy vs equations
  → Length: 100 words vs 2000

Same answer, different scores for different goals
  → Goal is "quick understanding" → short, clear answer scores high
  → Goal is "deep learning" → comprehensive answer scores high

Fix: define the scenario and audience before evaluating. Convert subjective judgment into a concrete scoring Rubric so LLM-as-Judge has clear criteria to follow.

Problem 3: Multi-Dimensional Quality

"AI responses should be good" unpacks into multiple independent dimensions that a single score can't capture:

Accuracy:     content matches facts, no hallucinations
Relevance:    the response addresses the user's actual question
Completeness: covers the core aspects of the question
Clarity:      well-organized, easy to follow
Helpfulness:  the user can actually act on this response

A response can be accurate but incomplete, or complete but unclear. Compressing 5 dimensions into 1 number loses too much information.

Four Evaluation Methods

No evaluation method works for every situation. Each carries costs. The choice depends on the scenario and budget.

Method            Accuracy   Speed      Cost   Best for
──────────────────────────────────────────────────────────────
Human evaluation  Highest    Slow (days) High   Baseline setup, periodic sampling
Rule/auto metrics Medium     Fast (ms)   Low    Format checks, keyword recall
LLM-as-Judge      High       Medium (min) Med   Main workhorse; needs bias mitigation
A/B testing       High       Slow (weeks) High  Production with real user behavior

Human Evaluation

Strengths: Highest accuracy, captures subtle quality differences, serves as the baseline for all other methods.

Weaknesses: Slow, expensive, doesn't scale. An evaluator reviews 50-100 samples per day; 1000 samples takes two weeks.

How to use: Build a "gold test set" for each critical scenario (about 100 samples, human-labeled), and use it as the reference standard. Validate automatic evaluation methods by comparing their results against the gold set.

Rule / Auto Metrics

Strengths: Millisecond speed, near-zero cost, easy to integrate into CI.

Weaknesses: Limited coverage; can't assess semantic quality. BLEU/ROUGE measure surface-level word overlap, not whether the meaning is correct.

# Useful automatic metrics
def check_format(output: str, schema: dict) -> bool:
    """Does the output conform to the JSON Schema?"""

def check_length(output: str, min_words: int, max_words: int) -> bool:
    """Is the length in the expected range?"""

def check_no_hallucination_markers(output: str) -> bool:
    """Does the output contain known hallucination phrases?"""

# Not useful for semantic quality
rouge_score = rouge.compute(predictions=[output], references=[reference])
# ROUGE penalizes semantically equivalent rephrasings and rewards verbatim repetition

How to use: Layer 2 structural checks — format compliance, length validation, prohibited phrase filtering. Don't use for semantic quality judgment.

LLM-as-Judge

Strengths: Scales well, understands semantics, evaluates quality dimensions that require reasoning.

Weaknesses: Susceptible to biases (covered in detail in Article 03), costs about 1/10th of human evaluation but 100x more than rule methods.

JUDGE_PROMPT = """Evaluate the following AI response (1-5 per dimension):

Dimensions:
1. Accuracy: Is the content factually correct?
2. Relevance: Does it address the user's question?
3. Clarity: Is it easy to understand?

User question: {question}
AI response: {answer}

Return as JSON: {{"accuracy": int, "relevance": int, "clarity": int}}"""

How to use: The main method for semantic quality dimensions that automatic metrics can't cover.

A/B Testing

Strengths: Tests actual user behavior, closest to business value. User clicks, adoption, and conversion are more trustworthy than model self-assessment.

Weaknesses: Requires sufficient traffic (statistical significance typically needs hundreds of exposures), takes weeks, and requires online experiment infrastructure.

How to use: Validate large changes (new model version, Prompt rewrite) for business impact. Not suitable for rapid iteration during development.

Combining Methods by Layer

In practice, the three approaches compose by layer:

Development (rapid iteration):
  Rule checks (automatic) + LLM-as-Judge (sampling)
  → CI runs format checks on every commit
  → Daily sampling quality evaluation

Release (quality gate):
  LLM-as-Judge (full or large sample) + human spot-check of critical cases
  → Release only when quality score exceeds threshold

Production (continuous monitoring):
  A/B testing + real-time rule checks + periodic human sampling
  → Monitor quality drift, surface systemic problems

Summary

The determinism trap: traditional "assert" thinking doesn't work for AI evaluation — multiple samples with statistics is more reliable than a single assertion
Subjectivity isn't a blocker: subjective quality becomes operational through clear scenario definitions, audience specifications, and scoring rubrics
No single best method: rules are fast but narrow in coverage; human evaluation is accurate but expensive; LLM-as-Judge is the best cost-effectiveness ratio for the main workload; A/B testing provides the final commercial validation

The next article goes into practice: how to decompose the vague goal "AI responses should be good" into a measurable L1/L2/L3 three-layer metric system.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

MCP Series (08): Enterprise Governance — Registry, Routing, and Observability

WonderLab — Fri, 17 Jul 2026 02:07:28 +0000

Why Governance Becomes Necessary

Three MCP Servers are manageable from memory. Twenty require a system.

As the number of MCP Servers in an enterprise grows, predictable problems emerge:

A new engineer builds a Jira tool that already exists because there's no directory
An Agent calls the deprecated search_jira tool (v1.x) instead of the current search_issues (v2.x)
A tool call fails with no log — unclear whether the Server crashed or the arguments were wrong
Token costs spike with no visibility into which Server's which tool caused it

Registry solves discovery. Routing solves dispatch. Observability solves diagnosis.

MCP Registry

A Registry is the enterprise directory of MCP Servers: their locations, versions, capabilities, and owners.

# mcp-registry.yaml
servers:
  - id: jira-tools
    name: Jira Tools
    description: "Search, create, and update Jira tickets"
    version: "2.1.0"
    domain: engineering
    owner: "@team-platform"
    status: active
    transport: stdio
    command: python
    args: ["/opt/mcp/jira/server.py"]
    capabilities:
      tools: [search_issues, create_issue, update_issue]
      resources: [jira://projects, jira://sprint/current]
    metrics:
      monthly_calls: 4521
      avg_latency_ms: 180
      error_rate: 0.2%

  - id: github-tools
    name: GitHub Tools
    version: "1.5.0"
    domain: engineering
    owner: "@team-platform"
    status: active
    transport: stdio
    command: npx
    args: ["-y", "@modelcontextprotocol/server-github"]
    capabilities:
      tools: [create_pull_request, search_repositories, get_file_contents]

  - id: jira-tools-legacy
    name: Jira Tools (Legacy)
    version: "1.2.0"
    domain: engineering
    status: deprecated
    deprecation:
      reason: "Superseded by jira-tools v2.x; search_jira renamed to search_issues"
      migration_guide: "Replace search_jira with search_issues; argument structure unchanged"
      removal_date: "2026-10-01"
    capabilities:
      tools: [search_jira, create_jira_ticket]

The Registry solves three things:

Discovery: new Agents check the Registry to learn what's available — no relying on hallway conversations
Deprecation signaling: deprecated status plus a migration_guide gives Agent code a concrete migration path
Ownership: every Server has an owner — when something breaks, you know who to contact

Tool Routing Strategies

When an Agent needs to "search Jira tickets," how does it find the right Server? Four strategies:

Strategy 1: Static Configuration (simplest)

Declare all Servers directly in Agent settings:

{
  "mcpServers": {
    "jira": {
      "command": "python",
      "args": ["/opt/mcp/jira/server.py"]
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"]
    }
  }
}

Simple and predictable. Adding a new Server requires updating every Agent config file manually.

Strategy 2: Domain-Based Loading

Load only the Servers relevant to the current task:

DOMAIN_SERVERS = {
    "engineering": ["jira-tools", "github-tools", "gitlab-tools"],
    "data": ["postgres-readonly", "bigquery-tools"],
    "communication": ["slack-tools", "email-tools"],
}

def load_servers_for_task(task_type: str) -> list[dict]:
    domain = classify_task_domain(task_type)
    server_ids = DOMAIN_SERVERS.get(domain, [])
    registry = load_registry()
    return [
        s for s in registry["servers"]
        if s["id"] in server_ids and s["status"] == "active"
    ]

Reduces unnecessary Server startup overhead. Each Agent loads only the tools it needs.

Strategy 3: Embedding Routing (semantic matching)

The same approach as Skill Series Article 06 — embed Server descriptions, embed the user request, find the nearest neighbors:

def route_to_server(user_input: str, registry: list[dict]) -> list[str]:
    query_embedding = embedder.embed(user_input)
    scored = []
    for server in registry:
        desc_embedding = get_cached_embedding(server["id"], server["description"])
        score = cosine_similarity(query_embedding, desc_embedding)
        scored.append((server["id"], score))

    scored.sort(key=lambda x: x[1], reverse=True)
    return [s[0] for s in scored[:3] if s[1] > 0.6]

Good for 20+ Servers where new ones get added frequently. The same limitation applies as in Skill routing: Servers in the same domain cluster in embedding space. Use negative examples in descriptions to distinguish them.

Strategy 4: Hierarchical Routing (recommended)

Coarse-filter by domain first, then run embedding matching within the domain:

def hierarchical_route(user_input: str, registry: list[dict]) -> list[str]:
    # Layer 1: LLM classifies domain quickly
    domain = llm_classify_domain(user_input)   # "engineering" / "data" / ...

    # Layer 2: embedding match within domain
    domain_servers = [s for s in registry if s.get("domain") == domain]
    return embedding_route(user_input, domain_servers)

Observability: Langfuse Integration

Without Trace, MCP tool calls are a black box. When something fails, you don't know why. When latency spikes, you don't know where. When token costs grow, you don't know which tool caused it.

Three-Layer Trace Structure

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context

langfuse = Langfuse()

@observe(name="mcp_tool_call")
async def traced_tool_call(
    server_id: str,
    tool_name: str,
    arguments: dict,
    call_fn
) -> dict:
    langfuse_context.update_current_observation(
        input={"tool": tool_name, "arguments": arguments},
        metadata={
            "server_id": server_id,
            "server_version": get_server_version(server_id),
        }
    )

    t0 = time.perf_counter()
    try:
        result = await call_fn(tool_name, arguments)
        latency_ms = (time.perf_counter() - t0) * 1000

        langfuse_context.update_current_observation(
            output=result,
            metadata={"latency_ms": round(latency_ms, 2), "success": True}
        )
        return result

    except Exception as exc:
        latency_ms = (time.perf_counter() - t0) * 1000
        langfuse_context.update_current_observation(
            output={"error": str(exc)},
            metadata={"latency_ms": round(latency_ms, 2), "success": False},
            level="ERROR"
        )
        raise

Session-Level Trace

@observe(name="agent_session")
async def run_agent_session(user_input: str, session_id: str):
    langfuse_context.update_current_trace(
        session_id=session_id,
        user_id="user:alice",
        metadata={"task_type": "jira_query"}
    )
    result = await agent.run(user_input)
    langfuse_context.update_current_observation(output={"result": result})
    return result

Every tool call inside this session automatically attaches to the session trace.

What Trace Answers

Which tool call is slowest?
  → sort Spans by latency_ms

Which Server has the highest error rate?
  → group by server_id, count success=False fraction

Where is token spend concentrated?
  → LLM Span usage field, grouped by tool_name

What caused a specific Agent session failure?
  → search by session_id in Langfuse, expand the trace tree

Alert Rules

Wire key metrics into your alerting system (Grafana / Prometheus):

# Prometheus alerting rules
groups:
  - name: mcp_server_alerts
    rules:
      - alert: MCPServerHighErrorRate
        expr: mcp_tool_call_error_rate_5m > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MCP Server {{ $labels.server_id }} error rate {{ $value | humanizePercentage }}"

      - alert: MCPToolHighLatency
        expr: mcp_tool_call_p90_latency_ms > 5000
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Tool {{ $labels.tool_name }} P90 latency {{ $value }}ms"

      - alert: DeprecatedToolInUse
        expr: mcp_deprecated_tool_calls_total > 0
        for: 0m
        labels:
          severity: info
        annotations:
          summary: "Deprecated tool {{ $labels.tool_name }} called {{ $value }} times today"

Three-Level Governance Roadmap

Level 1 — Individual/small team (do now):

Create mcp-registry.yaml with id / version / owner / status for every Server
Each Server's README lists its tools and usage examples

Level 2 — Team sharing (1-2 weeks):

Registry in Git, changes go through PR review
Langfuse Trace on every tool call: log tool_name / latency / success at minimum
Error rate alerts configured

Level 3 — Enterprise governance (ongoing):

Hierarchical routing (domain classification + embedding selection)
Monthly Server health report (call volume, error rate, top consumers)
Formal deprecation process (announce → alert → 90-day grace period → remove)

Summary

Registry is the foundation of discovery: without it, 20 Servers rely on word-of-mouth, duplicate development and version confusion are inevitable
Match routing strategy to scale: static config for 3 Servers; hierarchical routing (domain filter + embedding selection) for 20+, avoiding the same-domain embedding confusion problem
Observability turns black boxes into records: Langfuse's session-level + tool-call-level Trace means failure root causes go from 'unknown' to 'found in 5 seconds on the Dashboard'

References

Langfuse MCP Integration
MCP Series planning document: MCP Knowledge Series Outline

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project of the Day (#125): Open Interpreter — Open-Source AI Coding Agent, Rewritten in Rust, Supporting Kimi, Qwen, and DeepSeek

WonderLab — Fri, 17 Jul 2026 02:06:09 +0000

Introduction

"A coding agent for open models — focused on getting the best performance out of low-cost models."

This is article #125 in the Open Source Project of the Day series. Today's project is Open Interpreter — the 66k-Star open-source AI coding agent, now rewritten in Rust, and one of the most important local agent platforms in the open-model era.

Open Interpreter's history has an interesting inflection point. It originally (2023) lit up GitHub as a Python project — one-line description: "run ChatGPT Code Interpreter on your local machine." The Python community version has since been handed off to a community fork (endolith/open-interpreter). The official team rewrote the entire project in Rust, based on OpenAI's Codex codebase, with a sharper focus on a specific question: how do you get Kimi K3, Qwen, DeepSeek, and other low-cost open models to perform at near-top-tier levels in agent scenarios?

The answer is the Harness system: different models use different agent prompting strategies, letting each model work in the way it does best.

What You'll Learn

The Harness system design: why different models need different agent prompting strategies
What the Rust rewrite brought: architecture and performance changes
Computer Use capabilities: browser automation and native UI control
ACP protocol: what Agent Communication Protocol is
Codex SDK compatibility layer: one-line model substitution
Positioning differences vs. Claude Code

Prerequisites

Familiarity with AI coding agent concepts
Experience using local or low-cost LLMs is helpful
Understanding how Claude Code, Codex, and similar tools work helps contextualize the Harness concept

Project Background

From Python to Rust: A Significant Transformation

Open Interpreter's evolution:

2023: Python version launches
  ↓  50k+ Stars within a year
  ↓  Concept validated, but Python has performance and deployment limits

2024-2025: Rust rewrite begins
  ↓  Based on OpenAI Codex codebase
  ↓  Focus: best agent performance for low-cost open models

Late 2025 / 2026: Rust version takes lead
  ↓  Python community version handed to endolith/open-interpreter
  ↓  Official pushes Rust v0.0.26 (July 2026)

This rewrite isn't just a technical upgrade — it's a product repositioning: from "run ChatGPT Code Interpreter locally" to "agent framework optimized for open-weight models."

Author / Team

Organization: openinterpreter
License: Apache-2.0
Stack: Rust 96.6% + Python 2.6% + TypeScript
Version: v0.0.26 (July 2026)

Project Stats

⭐ GitHub Stars: 66,000+
🍴 Forks: 5,700+
📦 Releases: 57
📄 License: Apache-2.0
🦀 Language: Rust 96.6%

The Harness System: Core Innovation

Harness is the most critical design concept in Open Interpreter.

What Is a Harness?

Different AI models are trained differently, have different prompt preferences, and use different tool-calling formats. Applying the same agent prompting strategy to every model typically produces poor results with some of them — not because the model is weak, but because the "usage pattern is wrong."

A harness is "the optimal way to use a specific model as an agent": the system prompt format, tool definition style, context compression strategy, and error recovery logic — all tuned for that particular model.

Open Interpreter ships multiple harnesses:

Harness	Target Models	Notes
`native`	Default	General-purpose
`claude-code`	Anthropic Claude	Claude agent format
`claude-code-bare`	Anthropic Claude	Stripped Claude format
`kimi-code`	Kimi/Moonshot	Provider-recommended format, default for Kimi
`qwen-code`	Alibaba Qwen	Optimized for Qwen series
`deepseek-tui`	DeepSeek	DeepSeek model format
`swe-agent`	General	SWE-bench style, software engineering focus
`minimal`	General	Minimal token consumption
`zcode`	—	—

The Kimi K3 example: When you launch with --model kimi, Open Interpreter automatically uses the kimi-code harness — the Moonshot AI-recommended, field-validated agent prompting format for Kimi models, implemented natively in Rust. The result is substantially better performance from Kimi K3 on coding tasks compared to using a harness optimized for Claude.

Switching Harnesses

Type /harness kimi-code or /harness qwen-code directly in the TUI — no restart required.

Computer Use

This is the capability that distinguishes Open Interpreter from many coding agents: it doesn't just write code and run commands — it can directly operate graphical interfaces.

Browser Automation

Integrates agent-browser to drive a real browser:

User: "Find the OpenMontage repo on GitHub, extract the first three paragraphs of the README"
          ↓
Open Interpreter launches browser
          ↓
Navigates to github.com
          ↓
Searches and locates target repository
          ↓
Extracts and returns content

Native UI Automation

Integrates trycua/cua for native application control:

macOS: controls UI elements via Accessibility API
Windows: uses Windows Automation interface
Screenshot, click, type — works on any desktop application

QA Skill: A built-in skill built on these two capabilities. The agent automatically tests Web app or desktop app UIs without writing test scripts.

ACP Protocol Support

ACP (Agent Communication Protocol) is an open standard enabling interoperability between different AI agent tools.

# Run as ACP agent
interpreter acp

Once launched, Open Interpreter acts as a standard ACP agent endpoint, callable from ACP-compatible editors (VS Code, etc.) or other agent orchestration systems. You can use Open Interpreter as the backend agent directly from your IDE, without switching to a separate Open Interpreter interface.

Codex SDK Compatibility

If you've built something with OpenAI's Codex SDK, one line of configuration switches the underlying engine to Open Interpreter:

{
  "codexPathOverride": "interpreter"
}

Existing Codex SDK projects can switch to using local or other open-source models without code changes.

Quick Start

Install:

# macOS / Linux
curl -fsSL https://www.openinterpreter.com/install | sh

# Windows
irm https://www.openinterpreter.com/install.ps1 | iex

After installation, run i or interpreter to launch the TUI.

Configure models:

# Use Kimi K3 (automatically enables kimi-code harness)
interpreter --model kimi

# Use DeepSeek
interpreter --model deepseek

# Use Qwen
interpreter --model qwen

# Use Claude
interpreter --model claude-opus-4-6

TUI commands:

/model     Switch model
/harness   Switch harness
/help      All commands

Config file (~/.openinterpreter/config.toml):

[agent]
model = "kimi-k3"
harness = "kimi-code"

[permissions]
allow_file_writes = true
allow_network = true

Positioning vs. Claude Code

This question is worth answering directly:

Dimension	Open Interpreter	Claude Code
Open source	✅ Apache-2.0	Client open source
Model support	Multi-model (Kimi, Qwen, DeepSeek, Claude...)	Claude only
Cost	Depends on model; free/low-cost models available	Anthropic API pricing
Deployment	Local, data stays on-device	Local CLI, model in cloud
Customization	Replaceable harnesses, extensible components	Extensions via CLAUDE.md etc.
Computer Use	Built-in browser + native UI automation	Limited bash tool execution

These aren't in competition — they're different choices. If you primarily use Claude and value a polished experience, Claude Code is the better fit. If you want low-cost models like Kimi K3, fully local operation, or GUI automation capabilities, Open Interpreter is the more appropriate choice.

Links and Resources

🌟 GitHub: openinterpreter/openinterpreter
🌐 Website: openinterpreter.com
📦 Community Python fork: endolith/open-interpreter

Conclusion

Open Interpreter has gone through a meaningful transformation: from "run ChatGPT Code Interpreter locally" to "Rust agent framework optimized for open-weight models."

The timing is right. In 2025-2026, Kimi K3, Qwen, DeepSeek, and similar low-cost open models have been rapidly expanding their capability ceiling, increasingly approaching closed top-tier models. But using Claude/GPT-optimized agent formats on these models often produces suboptimal results. The Harness system's value is precise here: each model gets the prompting strategy it does best with.

The Rust rewrite brings performance advantages. More importantly, it makes ACP protocol support and Codex SDK compatibility possible — transforming Open Interpreter from a standalone tool into infrastructure that integrates into a broader ecosystem.

66k Stars mostly accumulated in the Python era, but active Rust iteration (57 releases, latest v0.0.26) shows this project continues to be seriously developed.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

MCP Series (07): Enterprise Deployment — Security, Authentication, and Version Management

WonderLab — Thu, 16 Jul 2026 13:46:30 +0000

Three Gaps Between Local and Production

A local MCP Server takes one command: python server.py. Enterprise production adds three required problems:

Authentication: stdio mode has no auth — any process can connect. Production needs explicit identity verification.
Process supervision: a Python process that dies doesn't restart itself and generates no alert. Production needs a guardian and health checks.
Smooth upgrades: multiple Agent sessions may connect to the same Server simultaneously. Upgrading can't drop those connections.

Authentication Options

API Key (preferred for internal services)

Simplest option, suitable for service-to-service calls inside a corporate network:

import os
from mcp.server import Server

EXPECTED_API_KEY = os.environ.get("MCP_API_KEY")
if not EXPECTED_API_KEY:
    raise RuntimeError("MCP_API_KEY environment variable is required")

server = Server("jira-tools")

For HTTP transport (non-stdio), validate in middleware:

from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request

class ApiKeyMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        key = (request.headers.get("X-API-Key")
               or request.headers.get("Authorization", "").removeprefix("Bearer "))
        if key != os.environ["MCP_API_KEY"]:
            return Response("Unauthorized", status_code=401)
        return await call_next(request)

OAuth 2.0 (cross-org or user-level permissions)

Use when different users need access to different subsets of data (different Jira projects per user):

The MCP spec (2025) defines OAuth integration interfaces. The Host (Claude Desktop / Claude Code) acquires the OAuth token during user login and passes it to the Server on each MCP connection.

mTLS (high-security internal communication)

For finance, healthcare, or government scenarios with strict data security requirements — mutual certificate verification:

import ssl

ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/certs/server.crt", "/certs/server.key")
ssl_context.load_verify_locations("/certs/ca.crt")
ssl_context.verify_mode = ssl.CERT_REQUIRED  # require client cert

# pass ssl_context to HTTP server

Authentication selection:

Internal service calls (same network)         → API Key + network isolation
Cross-org or user-level permission needs      → OAuth 2.0
High security (finance, healthcare, gov)      → mTLS

Docker Deployment

Minimal Dockerfile

FROM python:3.12-slim

WORKDIR /app

# Separate dependency install (cache layer)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Don't run as root (security best practice)
RUN useradd -r -s /bin/false mcpuser
USER mcpuser

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \
    CMD python -c "import sys; print('healthy')" || exit 1

CMD ["python", "jira_server.py"]

Production Docker Compose

# docker-compose.prod.yml
version: "3.9"

services:
  jira-mcp:
    build: .
    image: jira-mcp-server:1.3.0
    restart: unless-stopped          # auto-restart on crash
    environment:
      - MCP_API_KEY=${MCP_API_KEY}   # injected from .env or Secrets
      - JIRA_URL=${JIRA_URL}
      - JIRA_TOKEN=${JIRA_TOKEN}
      - LOG_LEVEL=INFO
    volumes:
      - ./logs:/app/logs
    networks:
      - mcp-internal                 # internal only
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "256M"
        reservations:
          memory: "128M"
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

networks:
  mcp-internal:
    driver: bridge
    internal: true                   # no external network access

Key configuration points:

restart: unless-stopped: restart on crash; stay stopped only on manual stop
internal: true: Docker network with no outbound connection — only containers on the same network can reach the Server
Resource limits: prevent a buggy Server from consuming host machine resources
Log rotation: prevent log files from growing unboundedly

Process Supervision (non-Docker)

# /etc/systemd/system/jira-mcp.service
[Unit]
Description=Jira MCP Server
After=network.target

[Service]
Type=simple
User=mcpuser
WorkingDirectory=/opt/jira-mcp
ExecStart=/usr/bin/python3 /opt/jira-mcp/jira_server.py
Restart=always
RestartSec=5
Environment="MCP_API_KEY=your-key"
Environment="JIRA_TOKEN=your-token"
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

systemctl enable jira-mcp
systemctl start jira-mcp
journalctl -u jira-mcp -f    # live log stream

Multi-Version Coexistence and Smooth Upgrades

The Problem

Multiple Agent sessions connect to MCP Server v1.2. You need to release v1.3 (adds a new tool) without dropping those connections.

Strategy: Parallel Versions + Traffic Switch

# docker-compose.prod.yml (parallel versions)
services:
  jira-mcp-stable:
    image: jira-mcp-server:1.2.0     # current stable, serves existing connections
    restart: unless-stopped
    networks: [mcp-internal]

  jira-mcp-canary:
    image: jira-mcp-server:1.3.0     # new version, accepts new connections
    restart: unless-stopped
    networks: [mcp-internal]
    deploy:
      replicas: 1

Point the Host config at the canary version first. Monitor it. Then switch stable:

{
  "mcpServers": {
    "jira": {
      "command": "docker",
      "args": ["exec", "jira-mcp-canary", "python", "jira_server.py"]
    }
  }
}

Version Number Rules

MAJOR.MINOR.PATCH

MAJOR: breaking changes
  → Remove a tool, rename a tool, change required inputSchema fields
  → Agent code that depends on this tool must update in sync

MINOR: backward-compatible additions
  → Add a tool, add optional parameters, extend return fields
  → Existing Agent code continues working; new features opt-in

PATCH: behavior-preserving fixes
  → Bug fixes, performance improvements, logging changes
  → Transparent upgrade

Declare the version in Server code:

server = Server(
    "jira-tools",
    version="1.3.0"   # returned to Client in initialize response
)

Deprecation Process

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "search_jira":   # old tool name
        logger.warning(
            "Tool 'search_jira' is deprecated. "
            "Use 'search_issues' instead. Removing in v2.0.0."
        )
        return await _search_issues(arguments)  # delegate to new implementation

Keep the old tool name for 90 days, log usage, then remove it in the MAJOR version.

Security Design Checklist

Authentication and authorization

[ ] Credentials injected via environment variables — never hardcoded in code or image
[ ] Internal services use API Key; cross-org or user-level permissions use OAuth
[ ] HTTP transport validates in middleware layer, not in tool handlers

Network isolation

[ ] Docker network set to internal: true — Server has no direct outbound access
[ ] Only necessary ports exposed (stdio mode requires no open ports)
[ ] Multiple Servers isolated on separate Docker networks

Tool security

[ ] Tool inputs have type validation and range checks (Article 04)
[ ] High-risk tools (writes, external API calls) have audit logs
[ ] Server's filesystem access restricted to necessary directories

Operations

[ ] Logs go to stderr, structured format (JSON), with rotation configured
[ ] restart: unless-stopped or systemd supervision ensures availability
[ ] Resource limits (CPU/memory) prevent abnormal Server behavior from affecting the host

Summary

Match auth to the scenario: API Key works for internal services, OAuth for cross-org or user-level access, mTLS for regulated industries — don't over-engineer
Docker three-pack: restart: unless-stopped (crash recovery) + internal: true network (isolation) + resource limits (stability)
Smooth upgrades need parallel versions: MINOR versions are backward-compatible and can replace directly; MAJOR versions run old and new in parallel, letting Agent code migrate at its own pace

References

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project of the Day (#124): Destructive Command Guard — Intercept rm -rf Before Your AI Agent Runs It

WonderLab — Wed, 15 Jul 2026 02:16:31 +0000

Introduction

"AI agents occasionally run catastrophic commands — rm -rf ./src, git reset --hard, DROP TABLE users — destroying hours of uncommitted work instantly."

This is article #124 in the Open Source Project of the Day series. Today's project is Destructive Command Guard (dcg) — a Rust hook that intercepts dangerous commands before AI coding agents execute them.

You may have encountered this scenario: you ask Claude Code to clean up temporary files and it uses rm -rf to delete the entire src directory. Or you ask it to reset a Git state and git reset --hard wipes two hours of uncommitted work.

This isn't the AI model becoming dumb. AI agents understand what commands do, but they may underestimate the cost given your specific current state. The gap between "what the command does" and "what it means for this particular moment" is where accidents happen.

dcg's solution: put a gate before command execution, intercept known dangerous patterns, and return decision authority to the human.

What You'll Learn

dcg's four-stage processing pipeline: how it makes an interception decision in under a millisecond
Context awareness — the critical design: why grep "rm -rf" should not be blocked
Coverage of 50+ security packs
Heredoc scanning: handling python -c "os.remove()" style implicit commands
Three bypass mechanisms and their design intent
Claude Code PreToolUse hook integration

Prerequisites

Regular use of Claude Code, Cursor, or similar AI coding tools
Familiarity with basic shell commands and Git operations
Direct or secondhand experience with AI agents executing dangerous operations helps contextualize the value

Project Background

What Is dcg?

Destructive Command Guard (dcg) is a protection tool that hooks into the command execution layer of AI coding agents. Written in Rust, it integrates as a PreToolUse hook in Claude Code's execution flow.

Before a command is executed, it passes through dcg's filter. If the command matches a known dangerous pattern, dcg outputs an interception message and the command doesn't run. If determined safe, the command executes normally with under 1ms latency.

Author

Jeffrey Emanuel: Original Python concept, expanded Rust implementation (pack system, Heredoc scanning, context classification)
Darin Gordon: Initial Rust port and performance optimizations
License: MIT

Project Stats

⭐ GitHub Stars: 4,400+
🍴 Forks: 165+
📄 License: MIT
🦀 Language: Rust (Edition 2024, nightly)

Four-Stage Processing Pipeline

Every command from agent intent to actual execution passes through four stages:

AI agent intends to execute a command
        ↓
Stage 1: JSON Parsing
        Receive PreToolUse hook JSON payload
        Extract command string
        Invalid payload → fail-open (pass through)
        ↓
Stage 2: Command Normalization
        /usr/bin/git → git
        /opt/homebrew/bin/python3 → python3
        Eliminates false positives from absolute paths
        ↓
Stage 3: Quick Filter
        SIMD-accelerated substring scan
        Skips 99%+ of safe commands instantly
        Suspicious keywords → full analysis
        ↓
Stage 4: Pattern Matching
        Check allowlist first (safe patterns)
        Then check denylist (dangerous patterns)
        Deny → output block message + alternative suggestion
        Allow → command executes normally

Total latency: < 1ms (200ms absolute timeout, then fail-open)

Fail-open design principle: When dcg itself encounters errors (parse failure, timeout, unexpected panic), it chooses to pass the command through rather than block it. This ensures dcg's own bugs can't freeze your workflow — security protection is an additive layer, not a new failure point.

Default-On Protection

No configuration needed after installation. The following protection is immediate:

core.filesystem (permanent, cannot be disabled)

Blocks dangerous deletion operations outside temp directories:

# Will be blocked:
rm -rf ./src
rm -rf ~/Documents
find . -name "*.js" -delete

# Won't be blocked (legitimate operations):
rm -rf /tmp/build_cache
rm /tmp/test_output.log

core.git (default on)

Blocks Git operations that destroy commit history or uncommitted work:

# Will be blocked:
git reset --hard HEAD~5       # destroys uncommitted changes
git push --force              # overwrites remote history
git clean -fd                 # deletes untracked files
git rebase --root             # rewrites entire history

# Block message example:
# ════════════════════════════════════════════════
# BLOCKED  dcg
# ────────────────────────────────────────────────
# Reason:  git reset --hard destroys uncommitted changes
# Command: git reset --hard HEAD~5
# Tip:     Consider using 'git stash' first.
# ════════════════════════════════════════════════

system.disk (default on)

Blocks disk-level dangerous operations:

# Will be blocked:
mkfs.ext4 /dev/sda
dd if=/dev/zero of=/dev/sdb
fdisk /dev/sda

Context Awareness: The Critical Design

dcg must distinguish between "a command string appearing in data context" and "a command string in execution context":

# These should NOT be blocked:
grep "rm -rf" README.md         # searching for a string, not executing
cat ~/.bashrc | grep "reset"    # viewing config, not executing
echo "never run: rm -rf /"      # outputting a string

# These SHOULD be blocked:
rm -rf ./important_dir
$(rm -rf /)                     # dangerous command in command substitution
`git reset --hard`              # backtick execution

The context classification module analyzes the semantic role of the command — is it being passed as an argument to grep/echo/cat, or is it a standalone executable command? That determination drives the decision.

Heredoc and Inline Code Scanning

This is a capability most similar tools lack — scanning inline code passed to interpreters:

# These attack patterns are scanned:
python -c "import os; os.remove('/etc/passwd')"
python -c "import shutil; shutil.rmtree('./src')"
bash -c "rm -rf /var/log/*"
node -e "require('fs').rmdirSync('./dist', {recursive: true})"

# Heredoc form:
python3 << 'EOF'
import os
os.remove('/important/file')
EOF

dcg uses tree-sitter and ast-grep to parse inline code, finding dangerous operations inside it rather than doing naive string matching.

50+ Opt-In Security Packs

Beyond default protection, packs can be enabled as needed:

Databases:

[packs]
"db.postgres" = true    # DROP TABLE, TRUNCATE, DELETE without WHERE
"db.mysql" = true
"db.mongodb" = true
"db.redis" = true       # FLUSHALL, FLUSHDB
"db.sqlite" = true

Kubernetes:

"k8s.kubectl" = true    # delete namespace, force delete pods
"k8s.helm" = true       # helm delete, helm rollback

Cloud:

"cloud.aws" = true      # s3 rm --recursive, ec2 terminate-instances
"cloud.gcp" = true
"cloud.azure" = true

Docker:

"containers.docker" = true  # docker rm -f, docker system prune -a

Terraform:

"iac.terraform" = true  # terraform destroy, terraform state rm

Example block with db.redis enabled:

# Will be blocked:
redis-cli FLUSHALL             # wipes entire Redis instance
redis-cli FLUSHDB              # wipes current database

Three Bypass Mechanisms

dcg is bypassable by design. The goal is "add friction," not "make it impossible":

Method 1: Single-command bypass (temporary)

DCG_BYPASS=1 git reset --hard HEAD~5

The environment variable only affects this one execution. The same command will still be blocked next time.

Method 2: One-time exception

dcg allow-once <code>   # code from the authorization code in the block message

Appropriate when you need to execute one specific operation without affecting future blocking rules.

Method 3: Permanent allowlist

dcg allowlist add "git reset --hard HEAD"   # allow this specific command
dcg allowlist add "rm -rf ./dist"           # allow cleaning build directory

Written to config file, permanent.

Claude Code Integration

# One-line install (easy-mode auto-configures Claude Code hook)
curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/destructive_command_guard/main/install.sh?$(date +%s)" | bash -s -- --easy-mode

This registers a PreToolUse hook in Claude Code's configuration. From that point, every command Claude Code prepares to execute passes through dcg first.

Manual configuration (~/.claude/settings.json):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "dcg"
          }
        ]
      }
    ]
  }
}

CI Scan Mode

Beyond real-time interception, dcg has a scan command for auditing dangerous commands already committed to a repository:

# Scan current repository
dcg scan ./

# Scan specific file types
dcg scan ./ --include "*.sh,*.yml,Dockerfile,Makefile"

The scanner understands execution context:

RUN rm -rf /var/cache/* in Dockerfile (reasonable, inside build layer)
- run: kubectl delete namespace prod in GitHub Actions workflow (worth flagging)
clean: rm -rf ./dist in Makefile (reasonable, clear intent)
rm -rf $TMPDIR in shell script (check whether TMPDIR might have unexpected value)

Links and Resources

🌟 GitHub: Dicklesworthstone/destructive_command_guard
📦 Install: install.sh --easy-mode

Conclusion

Destructive Command Guard addresses a new risk category that comes with AI coding tool adoption: AI agents have a much broader operational scope than previous automation tools. They actually run shell commands, manipulate file systems, and execute Git operations. A misjudgment now causes much more damage than running a wrong command the old way.

dcg's design philosophy is "add friction rather than block completely." It doesn't prevent you from doing anything — it just inserts a confirmation step before dangerous operations. Fail-open design ensures it never becomes a failure point itself. Context awareness and Heredoc scanning keep the false positive rate low enough not to interrupt normal work.

The 50+ security pack coverage extends the protection well beyond the obvious "prevent rm -rf" use case, bringing production-level dangerous operations in databases, Kubernetes, and cloud services into the protection scope.

For engineers already using Claude Code or similar tools in daily development, the cost to install dcg is minimal. The safety margin it provides is worth having.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.

MCP Series (06): MCP vs Function Calling — A Data-Driven Selection Guide

WonderLab — Tue, 14 Jul 2026 01:57:31 +0000

Two Specific Questions

Article 01 covered MCP's architectural advantage: standardized tool reuse. Engineers making an actual selection ask two more concrete questions:

How large is the MCP process communication overhead? Does it affect user experience?
When does MCP's code size actually become smaller than Function Calling?

Real benchmark data answers both.

Benchmark Design

Test subject: the same "search issues" feature

Method A (Function Calling): tool definition and handler live in Agent code, executed as a direct Python function call
Method B (MCP): tool runs in a standalone Server process, called via stdio subprocess

20 calls each (5 warm-up excluded), recording P50/P90/mean.

Latency Results

Method                           Mean       P50       P90       Min
──────────────────────────── ────────  ────────  ────────  ────────
Direct function call            0.01ms     0.01ms     0.01ms    0.005ms
MCP stdio call                  2.09ms     2.05ms     2.37ms     2.00ms

MCP overhead per call: +2.08ms  (283x slower than direct)
MCP server startup (one-time):  570ms
Calls to amortize startup cost: ~274

283x sounds alarming. In context: LLM inference takes 5–30 seconds. Tool calls are one step in that chain. At this timescale, 2ms of protocol overhead is imperceptible.

When the 2ms overhead actually matters:

Typical Agent task: LLM inference 15s + tool call 2ms → 0.01% overhead
                                                        → negligible

Real-time chatbot: target < 200ms, tool call on critical path
                                                        → evaluate carefully

High-frequency automation: 100 tool calls/second
                           2ms × 100 = 200ms extra latency/sec
                                                        → evaluate carefully

The MCP Server initializes once per session and serves all tool calls after that. Sessions exceeding 274 tool calls (570ms ÷ 2.08ms) reach parity with Function Calling on total latency. Most Agent tasks surpass that threshold; the startup cost amortizes fast.

Code Size Results

Function Calling (definition + handler + Agent loop): 43 lines
MCP Server (standalone process, complete server):     32 lines
MCP Agent code (zero tool code):                       0 lines

At N=1 the gap is small. It grows with every additional project:

Projects sharing one tool    FC total lines    MCP total lines    MCP saves
──────────────────────────────────────────────────────────────────────────
N=1                               43                 32              11
N=2                               86                 32              54
N=3                              129                 32              97
N=5                              215                 32             183

Function Calling code scales linearly (every Agent maintains its own tool code). MCP code stays constant (Server written once). The reuse benefit compounds with each additional project.

Why 283x Is Not the Right Metric for Selection

The raw multiplier misleads. A tool call's actual cost has three components:

Total latency = LLM inference + tool execution + protocol overhead

Typical scenario:
  LLM inference:      8,000ms
  Tool execution:       200ms
  MCP protocol:           2ms
  ─────────────────────────────
  Total:              8,202ms

Function Calling: 8,200ms
MCP:              8,202ms

Difference: 0.024%

Protocol overhead disappears next to LLM inference time. Latency doesn't drive the selection decision.

Decision Framework

Core question: how many projects will use this tool?

Projects = 1, AND:
  Tool logic < 30 lines
  Rapid prototype, unclear if long-term
  → Function Calling

Projects >= 2, OR:
  Tool needs state (connection pool, auth session)
  Non-engineers need to install/configure it (Claude Desktop ecosystem)
  Team wants a shared, centrally maintained tool standard
  → MCP Server

MCP is wrong when:
  Real-time requirement < 10ms (API gateway, live recommendations)
  Tool logic is tightly coupled to Agent logic; separation adds nothing
  Session lifetime is very short (< 100 tool calls) and startup time matters

Selection logic in one diagram:

Tool reuse (N projects × M calls)
    │
    ├── Single project, infrequent ──────→ Function Calling (simpler)
    │
    ├── Multiple projects sharing ────────→ MCP Server (reuse pays off)
    │
    ├── Needs state ───────────────────────→ MCP Server (process holds state)
    │
    └── Real-time requirement < 10ms ────→ Function Calling (no overhead)

Side-by-Side Implementation

Function Calling (43 lines — tool code in every Agent):

# Tool schema
SEARCH_TOOL = {
    "name": "search_issues",
    "description": "Search Jira issues by keyword",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"]
    }
}

# Tool implementation
def search_issues(query: str) -> str:
    results = [i for i in ISSUES if query.lower() in i["summary"].lower()]
    return "\n".join(f"[{i['key']}] {i['summary']}" for i in results)

# Agent loop (every Agent needs this)
def run_agent(user_input: str):
    client = anthropic.Anthropic()
    messages = [{"role": "user", "content": user_input}]
    while True:
        response = client.messages.create(
            model="claude-sonnet-4-6", tools=[SEARCH_TOOL], messages=messages)
        if response.stop_reason != "tool_use":
            return response.content[0].text
        tc = next(b for b in response.content if b.type == "tool_use")
        result = search_issues(tc.input["query"])
        messages.extend([...])

MCP Server (32 lines once — zero tool code in Agent):

# jira_server.py (written once, reused by every Agent)
server = Server("jira-tools")

@server.list_tools()
async def list_tools():
    return [Tool(name="search_issues",
                 description="Search Jira issues by keyword",
                 inputSchema={"type": "object",
                              "properties": {"query": {"type": "string"}},
                              "required": ["query"]})]

@server.call_tool()
async def call_tool(name, arguments):
    query = arguments["query"].lower()
    results = [i for i in ISSUES if query in i["summary"].lower()]
    text = "\n".join(f"[{i['key']}] {i['summary']}" for i in results)
    return [TextContent(type="text", text=text)]

async def main():
    async with stdio_server() as (r, w):
        await server.run(r, w, server.create_initialization_options())

asyncio.run(main())

# Agent configuration: one line in settings.json — no tool code in Agent at all

Summary

Three conclusions from real numbers:

2ms overhead is negligible in Agent contexts: LLM inference is 8,000ms, protocol overhead is 2ms — 0.024% of total. "283x slower" is technically accurate but practically misleading
570ms startup cost needs a warm-up design: if the Agent must respond to the first tool call immediately, initialize the MCP Server at session start rather than on first use
Code reuse is the core reason to choose MCP: at N=1 the difference is small (43 vs 32 lines); at N=3 projects Function Calling needs 129 lines while MCP stays at 32 — the gap grows with every additional project and every additional tool

References

Full demo code: mcp-06-comparison

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Open Source Project of the Day (#123): Vibe-Trading — Quantitative Research in Natural Language, an AI-Powered Personal Trading Agent

WonderLab — Tue, 14 Jul 2026 01:55:31 +0000

Introduction

"Turn natural language finance questions into runnable analysis, backtests, and trading workflows."

This is article #123 in the Open Source Project of the Day series. Today's project is Vibe-Trading — an AI-driven quantitative trading research framework from HKUDS (Hong Kong University Data Science Lab).

Quantitative research has a high traditional barrier: you need to know which data API to use, how to handle missing data, which backtesting framework to choose, how to write strategy code, how to do statistical validation. Any of these steps can block progress.

Vibe-Trading's starting premise: describe in natural language what you want to research, and let the AI agent handle the technical details underneath. "Backtest a BTC-USDT 20/50 MA strategy" — one sentence, and the agent automatically selects a data source, fetches data, writes the strategy, runs the backtest, and generates a report.

This isn't a promise to make you money. It's a research tool, and its own security design makes that clear: read-only defaults, no fund custody, all live trading relays through broker interfaces.

What You'll Learn

Natural language → executable strategy workflow: Plan/Ground/Execute/Validate/Deliver
19 free data sources and intelligent fallback chain design
461 pre-built alpha factors (Alpha Zoo)
Shadow account: analyzing your own trading behavior from historical trade records
Multi-agent investment committee mode
Security design: AST sandbox, no fund custody principle

Prerequisites

Basic understanding of quantitative trading concepts (strategy, backtest, alpha factor)
Python basics
Trading experience in A-shares, US stocks, or crypto helps contextualize use cases

Project Background

What Is Vibe-Trading?

Vibe-Trading is an open-source quantitative research workbench that uses AI agents to translate natural language research intent into executable analysis code and backtesting workflows.

Its positioning is not "AI that trades for you" but "lowering the technical barrier to quantitative research" — making professional quant tools accessible without requiring software engineering as a prerequisite.

Author / Team

Lab: HKUDS (Hong Kong University Data Science Lab)
License: MIT
Language: Python 3.11+

Project Stats

⭐ GitHub Stars: 21,800+
🍴 Forks: 3,800+
🛠 Agent tools: 68
📊 Alpha factors: 461
📡 Data sources: 19 free + 63+ premium
📄 License: MIT

Five-Stage Workflow

Plan

The agent parses the natural language query and decides:

Which skills to invoke
Which tools to call
Which data sources to connect
Whether to activate multi-agent collaboration

Ground

Market data fetched via fallback chains:

A-shares (ordered by ascending IP-ban risk):

tencent → mootdx → eastmoney → baostock → akshare → tushare

US equities:

yahoo → stooq → sina → yfinance → finnhub → alphavantage

Crypto:

okx → ccxt → yfinance

One get_market_data call — upper-layer code doesn't need to know which source is used; the data layer handles fallbacks transparently.

Execute

The agent generates strategy code and runs backtest engines inside an AST sandbox. Supported engines:

Market	Engine
A-shares	China A-Share Engine
US equities	US Equity Engine
HK equities	HK Equity Engine
India	India NSE/BSE Engine
Crypto	Crypto Engine
China futures	China Futures Engine
Global futures	Global Futures Engine
Forex	Forex Engine
Options	Options Engine
Mixed portfolio	CompositeEngine

Validate

Statistical validation methods:

Monte Carlo simulation
Bootstrap confidence intervals
Walk-Forward testing
Strategy run cards

Deliver

Output formats:

HTML / PDF reports
Pine Script (TradingView)
TDX (China-specific trading software)
MetaTrader 5 MQL5

Core Features

Self-Improving Agent

The agent's memory and skill system:

Cross-session persistent memory: Previously researched strategies, discovered patterns, and stated hypotheses are preserved
87 skills across 9 categories: Quantitative research, factor analysis, risk management, portfolio optimization, and more
5-layer context compression: Handles context expansion in long research sessions
Automatic skill evolution: Summarizes new skills during research, expanding the capability library

Alpha Zoo (Factor Library)

461 pre-built quantitative factors, benchmarkable in one command:

# Benchmark top 20 factors from GTJA191, CSI300 universe, 2018-2025
vibe-trading alpha bench \
  --zoo gtja191 \
  --universe csi300 \
  --period 2018-2025 \
  --top 20

Output: IC mean, IC standard deviation, IR, and IC-positive ratio per factor.

Supported factor libraries:

Qlib158: Classic factor set from Microsoft's open-source quant library
Kakushadze101: The classic 101 alpha factors from academic literature
GTJA191: Guotai Junan's 191 quantitative factors
PIT-safe fundamental factors: Fundamental data factors designed to avoid Point-in-Time bias

Shadow Account

One of Vibe-Trading's most interesting features: upload your historical trade records and let AI analyze your behavioral patterns.

# Upload broker export
vibe-trading --upload trades_export.csv

# Analyze trading behavior
vibe-trading run -p "Analyze my trade history, identify behavioral biases, extract my actual trading strategy"

Diagnosed bias types:

Disposition Effect: Selling winners too early, holding losers too long
Overtrading: High frequency with no corresponding alpha
Momentum Chasing: Buying after run-ups, selling after drawdowns
Anchoring Bias: Over-relying on specific price reference points

Output:

Quantified bias magnitude report
Extracted "shadow strategy" (your actual behavioral pattern)
Comparison against backtest baseline
HTML/PDF audit report

Multi-Agent Investment Committees

For complex research tasks, multiple agents can run in parallel:

Mode	Composition
Investment committee	Analyst, risk officer, portfolio manager in parallel
Quant desk	Factor research, strategy development, backtest validation in division of labor
Crypto team	On-chain analysis, DEX liquidity, macro hedging
Risk committee	Multi-dimensional risk assessment and stress testing

Agents work in parallel with streamed progress output, results consolidated into a comprehensive final report.

Quick Start

Install:

pip install vibe-trading-ai

# All 16 IM channel adapters (Telegram, Slack, Discord, WeChat, Feishu, etc.)
pip install "vibe-trading-ai[channels]"

# Telegram only
pip install "vibe-trading-ai[telegram]"

Basic usage:

# Strategy research and backtest
vibe-trading run -p "Backtest a BTC-USDT 20/50 MA crossover strategy for 2024"

# Macro analysis
vibe-trading run -p "Analyze current US tech stock valuations compared to 2000 dot-com levels"

# Factor research
vibe-trading run -p "Test momentum factor effectiveness in A-shares, 2015-2025"

# Shadow account analysis
vibe-trading --upload broker_export.csv
vibe-trading run -p "Identify my behavioral biases from the last 2 years of trading"

Python API:

from vibe_trading import Agent

agent = Agent(provider="anthropic", model="claude-opus-4-6")

result = await agent.run(
    "Analyze CSI300 monthly momentum effect, backtest 2015-2025, "
    "output annualized return, Sharpe ratio, and maximum drawdown"
)

Docker:

docker compose up -d
# WebUI: http://localhost:3000

Broker Connections

Live and paper trading via 10 broker integrations:

Market	Brokers
US equities	Robinhood, Alpaca, Interactive Brokers
A-shares / HK	Futu, Longbridge, Tiger Brokers
Crypto	OKX, Binance
India	Dhan, Shoonya

Critical safety design:

No fund custody — system only relays trade intent, brokers execute
Brokers without structural paper/live separation capped at read-only
Filesystem-level kill switch for emergency stop
Complete operation audit ledger

Security Design

Vibe-Trading has done substantial security work — necessary for any system that executes AI-generated code:

AST Sandbox: Backtest code runs inside a hardened AST sandbox:

Blocked at compile time:
  × network calls (no external network access)
  × subprocess (no spawning subprocesses)
  × eval/exec (no dynamic code execution)
  × os.environ (no environment variable access)
  × file writes outside designated directories

AI-generated strategy code cannot perform any of these dangerous operations regardless of what the code contains.

Additional security measures:

CSRF protection + SSRF defenses
Rate limiting
Docker multi-stage builds with digest-pinned images
Short-lived single-use SSE authentication tickets

Links and Resources

🌟 GitHub: HKUDS/Vibe-Trading
📦 PyPI: vibe-trading-ai

Conclusion

Vibe-Trading represents a direction: lowering the technical barrier to quantitative research from "mastering multiple specialized tools" to "being able to describe what you want to research."

461 pre-built factors, 19 free data sources, backtest engines for 10 markets — this technical depth isn't a feature list. It's pre-solving the integration problems that anyone doing quantitative research has to work through, so users can focus on the research itself.

The shadow account feature is rare and genuinely useful. Most retail investors have accumulated trade records they've never systematically analyzed. This feature lets you compare "how I think I trade" with "how I actually trade."

The security design (AST sandbox, no fund custody, read-only defaults) shows the team understands where this class of system's risks lie and built defenses at the right places.

21.8k Stars from an academic background team at HKUDS signals both academic credibility and serious engineering.

Disclaimer: This article describes an open-source technical tool and does not constitute investment advice. Historical backtest performance does not guarantee future returns.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.