DEV Community

Cover image for Stop false consensus in multi-agent swarms
Screwtape.ai
Screwtape.ai

Posted on Originally published at screwtape.ai

Stop false consensus in multi-agent swarms

#llm #ai #python #agents

Majority vote is the default way to reconcile a multi-agent run. Three agents answer, two or three agree, and the orchestrator moves on. The trouble is that agents in the same swarm often share a prompt, a retrieved document, or each other's messages. When they agree, they may just be repeating the same mistake. That's false consensus: high agreement with little independent evidence behind it.

This post shows how to check for it with Screwtape.ai, a judge-agent API with two endpoints:

Endpoint You send You get
POST /v1/judge task, agent_output (required), evidence[], criteria[], retain (optional) verdict (pass \
POST /v1/consensus task, answers[] (min 2, each {agent_id?, output}), retain (optional) consensus_answer, agreement_score, false_consensus_risk (low \

Auth is Authorization: Bearer <key>. Free keys come from screwtape.ai/signup and include 100 calls per UTC day. Without a key, each call can be paid with an x402 PAYMENT-SIGNATURE header: $0.005 USDC on Base (eip155:8453).

1. The curl

This is the request from the OpenAPI grounded example, field for field:

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size and launch site.","agent_output":"The crew size was 4. The mission launched from Cape Canaveral.","evidence":["The mission launched on 2024-03-01 from Cape Canaveral.","The crew size was 4."],"criteria":["crew size","Cape Canaveral"]}'
Enter fullscreen mode Exit fullscreen mode

Real response with no key (I ran this on 2026-10-20 without an Authorization header). The API answers 402 and describes the x402 payment it accepts:

{"error":"payment_required","message":"PAYMENT-SIGNATURE header is required. Send a Bearer API key, or pay per call with x402.","request_id":"97bf7d4e-4930-41eb-88ce-89c9e75dcd65","x402":{"x402Version":2,"error":"PAYMENT-SIGNATURE header is required. Send a Bearer API key, or pay per call with x402.","resource":{"url":"https://screwtape.ai/v1/judge","description":"One Screwtape.ai judge or consensus call","mimeType":"application/json","serviceName":"Screwtape.ai"},"accepts":[{"scheme":"exact","network":"eip155:8453","amount":"5000","asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","payTo":"0x028Bb9A57B8C27c22f6B39dd00e7eFF1FB60344B","maxTimeoutSeconds":300,"extra":{"name":"USD Coin","version":"2","assetTransferMethod":"eip3009"}}],"extensions":{}}}
Enter fullscreen mode Exit fullscreen mode

With a key, the 200 response has this shape. This is the example published on the screwtape.ai homepage for the deterministic engine. It's an example, not a call I ran:

{
  "verdict": "pass",
  "score": 0.9,
  "reasons": [
    "2 of 2 claims are grounded in the supplied evidence.",
    "2 of 2 criteria matched the output.",
    "Deterministic string checks only. No model was called."
  ],
  "engine": "deterministic",
  "retention": { "retained": false },
  "request_id": "8d0c2f1e-6b4a-4e3a-9c1d-2f6a0b7e5d44"
}
Enter fullscreen mode Exit fullscreen mode

When the service has a model key configured, engine is llm, and the default model is openai/gpt-oss-120b. The wording of the reasons will differ.

2. Catching copied agreement 

curl -s https://screwtape.ai/v1/consensus \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size.","answers":[{"agent_id":"alpha","output":"The crew size was 4."},{"agent_id":"beta","output":"The crew size was 4."},{"agent_id":"gamma","output":"The crew size was 4."}]}'
Enter fullscreen mode Exit fullscreen mode

Example shape, from the homepage (not executed here):

{
  "consensus_answer": "The crew size was 4.",
  "agreement_score": 1,
  "false_consensus_risk": "high",
  "reasons": [
    "Answers are nearly verbatim. High agreement may be copied rather than independent.",
    "Agreement is token overlap, not a model judgment."
  ],
  "request_id": "..."
}
Enter fullscreen mode Exit fullscreen mode

agreement_score: 1 together with false_consensus_risk: "high" is the point. A majority vote would accept this answer immediately. Screwtape tells you the agreement looks copied, so you can bring in an independent check before you trust it.

3. A shared client (stdlib only)

All three framework examples import this file. Note the explicit User-Agent: the Cloudflare edge in front of screwtape.ai returned 403 to Python's default Python-urllib/x.y agent when I tested.

# screwtape_client.py
"""Minimal Screwtape.ai HTTP client (stdlib only). Spec: https://screwtape.ai/openapi.json"""
import json
import os
import urllib.request

BASE = "https://screwtape.ai"


def _post(path: str, body: dict) -> dict:
    req = urllib.request.Request(
        BASE + path,
        data=json.dumps(body).encode(),
        headers={
            "Authorization": f"Bearer {os.environ['SCREWTAPE_API_KEY']}",
            "Content-Type": "application/json",
            # Cloudflare in front of screwtape.ai returns 403 to the default
            # "Python-urllib/x.y" User-Agent, so send an explicit one.
            "User-Agent": "screwtape-example/1.0",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)


def judge(task: str, agent_output: str, evidence=None, criteria=None) -> dict:
    """POST /v1/judge -> {verdict: pass|fail|retry, score, reasons, request_id, ...}"""
    body = {"task": task, "agent_output": agent_output}
    if evidence:
        body["evidence"] = evidence
    if criteria:
        body["criteria"] = criteria
    return _post("/v1/judge", body)


def consensus(task: str, answers: list[dict]) -> dict:
    """POST /v1/consensus. answers: [{"agent_id": "...", "output": "..."}], at least 2.
    -> {consensus_answer, agreement_score, false_consensus_risk: low|medium|high, reasons, request_id}"""
    return _post("/v1/consensus", {"task": task, "answers": answers})
Enter fullscreen mode Exit fullscreen mode

4. CrewAI: a task guardrail

CrewAI tasks accept a guardrail callable that takes a TaskOutput and returns (bool, Any). If it returns False, the second value goes back to the agent as feedback, and the task retries up to guardrail_max_retries times.

# crewai_guardrail.py
# pip install crewai   (tested against crewai 1.15.x)
# Needs SCREWTAPE_API_KEY plus whatever LLM your crew uses (default: OPENAI_API_KEY).
from crewai import Agent, Crew, Task
from crewai.tasks.task_output import TaskOutput

from screwtape_client import judge

EVIDENCE = [
    "The mission launched on 2024-03-01 from Cape Canaveral.",
    "The crew size was 4.",
]


def screwtape_guardrail(output: TaskOutput) -> tuple[bool, str]:
    result = judge(
        task="State the crew size and launch site.",
        agent_output=output.raw,
        evidence=EVIDENCE,
        criteria=["crew size", "Cape Canaveral"],
    )
    if result["verdict"] == "pass":
        return True, output.raw
    # On fail/retry, CrewAI sends this feedback back to the agent and retries.
    return False, "Screwtape verdict: {}. {}".format(result["verdict"], " ".join(result["reasons"]))


writer = Agent(
    role="Mission summarizer",
    goal="Answer only from the supplied facts.",
    backstory="You summarize mission briefs without adding details.",
)

task = Task(
    description="Facts: " + " ".join(EVIDENCE) + " State the crew size and launch site.",
    expected_output="Two short sentences.",
    agent=writer,
    guardrail=screwtape_guardrail,
    guardrail_max_retries=2,
)

if __name__ == "__main__":
    print(Crew(agents=[writer], tasks=[task]).kickoff().raw)
Enter fullscreen mode Exit fullscreen mode

5. LangGraph: route on false-consensus risk

The consensus check is a node, and a conditional edge routes on false_consensus_risk. The agent node is a stub, so this graph runs with only a Screwtape key.

# langgraph_consensus.py
# pip install langgraph   (tested against langgraph 1.2.x)
# Needs only SCREWTAPE_API_KEY. The three "agents" are stubs; swap in your LLM calls.
from typing import TypedDict

from langgraph.graph import END, START, StateGraph

from screwtape_client import consensus

TASK = "State the crew size."


class State(TypedDict, total=False):
    answers: list[dict]
    verdict: dict


def agents(state: State) -> State:
    # Replace with real agent calls. Identical answers are what false consensus looks like.
    outs = {"alpha": "The crew size was 4.", "beta": "The crew size was 4.", "gamma": "The crew size was 4."}
    return {"answers": [{"agent_id": k, "output": v} for k, v in outs.items()]}


def score(state: State) -> State:
    return {"verdict": consensus(TASK, state["answers"])}


def route(state: State) -> str:
    return "escalate" if state["verdict"]["false_consensus_risk"] == "high" else "accept"


def accept(state: State) -> State:
    print("accepted:", state["verdict"]["consensus_answer"])
    return {}


def escalate(state: State) -> State:
    print("needs independent check:", state["verdict"].get("reasons"))
    return {}


g = StateGraph(State)
g.add_node("agents", agents)
g.add_node("score", score)
g.add_node("accept", accept)
g.add_node("escalate", escalate)
g.add_edge(START, "agents")
g.add_edge("agents", "score")
g.add_conditional_edges("score", route, {"accept": "accept", "escalate": "escalate"})
g.add_edge("accept", END)
g.add_edge("escalate", END)
app = g.compile()

if __name__ == "__main__":
    app.invoke({})
Enter fullscreen mode Exit fullscreen mode

6. AutoGen: use Screwtape's MCP tools

Screwtape exposes judge and consensus over MCP Streamable HTTP at https://screwtape.ai/mcp, using the same auth and quota as REST. AutoGen's mcp_server_tools turns them into agent tools.

# autogen_mcp.py
# pip install "autogen-agentchat" "autogen-ext[mcp,openai]" "mcp<2"
# (tested with autogen 0.7.5; autogen-ext 0.7.5 fails to import with mcp 2.x, hence the pin)
# Needs SCREWTAPE_API_KEY and OPENAI_API_KEY.
import asyncio
import os

from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.ui import Console
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.tools.mcp import StreamableHttpServerParams, mcp_server_tools


async def main() -> None:
    server = StreamableHttpServerParams(
        url="https://screwtape.ai/mcp",
        headers={"Authorization": f"Bearer {os.environ['SCREWTAPE_API_KEY']}"},
    )
    tools = await mcp_server_tools(server)  # -> judge, consensus
    agent = AssistantAgent(
        name="reviewer",
        model_client=OpenAIChatCompletionClient(model="gpt-4o-mini"),
        tools=tools,
        system_message=(
            "Before you accept another agent's answer, call the judge tool with the task, "
            "the answer as agent_output, and any evidence. Report the verdict and reasons."
        ),
    )
    await Console(agent.run_stream(task=(
        "Task: State the crew size and launch site. "
        "Answer from worker agent: 'The crew size was 5. The mission launched from Vandenberg.' "
        "Evidence: 'The mission launched on 2024-03-01 from Cape Canaveral.', 'The crew size was 4.'"
    )))


if __name__ == "__main__":
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

You can call tools/list without a key, so discovery works before you sign up. tools/call needs a Bearer key or an x402 payment.

What it doesn't do

  • It doesn't run locally. It's a hosted API.
  • In deterministic mode, agreement is token overlap and grounding is claim-vs-evidence token matching. It isn't semantic reasoning. The reasons say so.
  • Judged content is kept in memory only, unless you send retain: true with an API key. Retained rows expire after 24 hours (privacy).

Try it

Get a free key (100 calls/day): https://screwtape.ai/signup. Then read the API reference or the raw OpenAPI.

This article was created with the help of AI by the Screwtape.ai team.

Top comments (0)