DEV Community: bayze-mcp

We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed

bayze-mcp — Fri, 06 Nov 2026 13:43:41 +0000

Three months ago bayze-mcp was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships five multi-model collaboration patterns — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included.

The five patterns, with real costs

Pattern	Mechanism	Measured cost	Use when
Cross-vendor review	A drafts → B structured-review gate → C revises	~$0.001/run	Default quality tier
Parallel + judge	N models answer → judge picks by hard constraints	$0.0013/run	Tasks with length/format caps
MoA aggregation	N models answer → synthesizer merges the best of each	$0.006/run	Open-ended generation
Revision loop	Review bounces → revise → re-review → escalate to a stronger model when exhausted	$0.002/escalation	Unstable first drafts
Debate	Two debaters open independently → K rebuttal rounds → arbiter verdict	$0.02–0.04/run	Analysis / decisions / risk assessment

The single most useful finding: pattern-task fit beats model quality. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both.

Nobody counts their own words

Five samples across six rounds:

Qwen-plus self-reported "exactly 10 words" — actual: 14
Qwen-plus self-reported "about 580 characters" — actual: 799
Claude haiku under-reported by 1–38 chars in earlier runs

No model's self-count is trustworthy. So bayze treats length like memory safety: the engine counts, deterministically — every output passes max_chars / max_words checks, and the model's opinion is never consulted.

Six rounds of debate testing: a bug-finding story

We stress-tested strategy: debate with a real question: "Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?"

Rounds 1–2, two invisible bugs. The test agent complained "my constraints are being ignored." They were — but not by the models: the context field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, maxOutputTokens: 2000 allowed ~3000 characters of output while a max_chars: 2000 gate rejected at 2000 — the engine was permitting output it was guaranteed to reject.

Round 3, the debate actually worked. Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run.

Rounds 4–5, engineering answers to "vibes" problems. Models treated "opening / rebuttal / verdict" discipline as their assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by self-calibration: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen fabricated "we audited this team's real invoices (redacted)" plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict.

Round 6, the last design flaw. Both models ended up on the same side ("worth migrating"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional stances: [pro, con], with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.

What the arbiter actually concluded

Condensed from two debates plus verdicts:

Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using "we can leave" as negotiating leverage.

Caveat that matters more than the conclusion: every dollar figure both models produced was fabricated (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible.

Try it

npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml

GitHub (bilingual README): https://github.com/caijianxiong2007-prog/bayze-mcp
Gitee mirror: https://gitee.com/collection-master/bayze-mcp
npm: https://www.npmjs.com/package/bayze-mcp
Every pattern's design trade-offs are recorded as ADR D1–D8 in docs/adr/0006-multi-ai-collaboration.md

MIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.

Bayze: A Local-First MCP Server for Cross-Vendor AI Model Review (Draft - Review - Revise)

bayze-mcp — Thu, 05 Nov 2026 08:15:08 +0000

The Problem

You're using an AI assistant (Cherry Studio, Claude Desktop, Cursor) and you want a second opinion on its answer. Today, that means manually copy-pasting between ChatGPT, Claude, and DeepSeek.

Bayze eliminates that friction. It's a local-first MCP server that turns "cross-vendor AI review" into a single tool call your AI client can make.

How It Works

Bayze runs a waterfall workflow: one model drafts, a second model reviews (with structured JSON verdict), a third model revises only when rejected.

✓ draft   deepseek-chat    2.1s  ~$0.001
✓ review  qwen-plus        2.7s  ~$0.001
    verdict: revise — "missing TLS handshake details"
✓ revise  doubao-lite       3.6s
  → Final output incorporates reviewer's corrections

Every step is auditable: which model, how many tokens, what cost, why it escalated.

Quick Start

npx -y bayze-mcp mcp --config your-config.yaml

Then add it to your MCP client (Cherry Studio, Claude Desktop, Cursor — any stdio-compatible client).

1. Configure providers (YAML)

providers:
  deepseek:
    kind: openai-compatible
    base_url: "https://api.deepseek.com/v1"
    api_key_ref: "env:BAYZE_DEEPSEEK_API_KEY"
    models:
      - id: deepseek-chat
        pricing: { currency: CNY, input_per_1m_tokens: 2.0, output_per_1m_tokens: 8.0 }

  dashscope:
    kind: openai-compatible
    base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
    api_key_ref: "env:BAYZE_DASHSCOPE_API_KEY"
    models:
      - id: qwen-plus

2. Define a workflow

workflows:
  review_then_revise_cn:
    strategy: waterfall
    limits:
      max_steps: 3
      total_timeout_ms: 120000
      max_cost_usd: 0.20
    steps:
      - name: draft
        provider: deepseek
        model: deepseek-chat
        input: original
        evaluate: [non_empty]
      - name: review
        provider: dashscope
        model: qwen-plus
        input: review_template
        run_if: always
        gate: structured_review
        review_schema: [verdict, reasons]
      - name: revise
        provider: ark
        model: doubao-seed-2-1-lite
        input: revision_template
        run_if: review_verdict_is_revise

3. Call it from your AI client

Just say: "Use review_then_revise_cn to process: [your task]"

Key Features

Feature	What it does
Structured review gate	Reviewer must return JSON `{"verdict":"accept\
Budget guard	Pre-call estimation, atomic reservation, fail-closed on unknown prices
Dual access mode	BYOK (your keys) or Bayze unified API (managed)
Auditable journal	Every step records model, tokens, cost, latency — never stores prompts
Privacy	Env-only keys, no telemetry, no cloud components, no listening ports
Deterministic checks	{% raw %}`max_chars`, `non_empty` — engine-level, not model-dependent

Supported Providers

OpenAI-compatible: DeepSeek, Qwen (DashScope), Doubao (Ark), OpenAI, any /v1 endpoint
Anthropic: Native Messages API adapter (different protocol, auth, error taxonomy)
Bayze unified API: Managed gateway with 53+ models
Mock: 13 scenarios for testing without keys

Real-World Results

Verified with 15 real cross-vendor runs across 7 providers:

Reviewer correctly identified missing TLS handshake details in a draft
Reviser incorporated reviewer's corrections (verified character-by-character)
Cost per 3-step review: ~$0.003 (less than a penny)
Latency: 8-15s for full waterfall