DEV Community

Weave
Weave

Posted on

How to Preview a Weave Router Benchmark Run Before Launch

#testing #python #ai #tutorial

A benchmark launch can fan out across tasks, attempts, and model routes. Before spending model or sandbox time, inspect the exact plan.

Weave Router's public weave-bench CLI has a --dry-run option for that step. It resolves the benchmark manifest, builds one Harbor command per comparison arm, prints the commands and version pins, then exits before launching any Harbor jobs. In this walkthrough, the plan selects one Terminal-Bench task and two arms. It produces no benchmark result.

Install the benchmark CLI

The harness requires Python 3.12 or newer and uv. This guide is pinned to Router commit cbe7584, so the source and manifest match the behavior described here:

git clone https://github.com/weave-os/router.git
cd router
git checkout cbe758442460ca68db8f4fe9837db087c50cf1e5
cd bench
uv sync --locked --extra harbor
Enter fullscreen mode Exit fullscreen mode

Cloning the repository and syncing dependencies use the network. After that setup is complete, the planning command below builds and prints a local plan; it does not call the Router or model APIs or start Harbor jobs.

Print a one-task plan

This command compares the Router's router arm with a direct GPT-5.6 Luna control, using the benchmark's pinned smoke task:

uv run --locked --extra harbor weave-bench run terminal-bench-4 \
  --arms router,luna \
  --smoke \
  --dry-run
Enter fullscreen mode Exit fullscreen mode

Representative output:

# terminal-bench-4 run_id=<generated-run-id>
# tasks=1 (manifest has 66); harbor=0.22.0 (published: 0.22.0) codex=0.150.0
...
--include-task-name terminal-bench/cad-model
Enter fullscreen mode Exit fullscreen mode

The CLI generates the run ID. The important details are the selected task and the pinned versions, followed by the Harbor command for each arm.

What the plan means

  • One task, not 66: --smoke selects terminal-bench/cad-model from the 66-task manifest.
  • Two arms, two Harbor jobs: router sends requests through Weave Router's /beta policy; luna calls GPT-5.6 Luna directly. This is not a same-model comparison because the Router policy chooses a model per request.
  • Two attempts per arm: the pinned configuration sets two attempts and up to 12 concurrent tasks per job. For this one-task example, a real run is configured for up to four task attempts in total. An attempt can make multiple model requests, so that number is not an API-call count or a cost estimate.
  • No credentials in the plan: dry-run output shows the names of environment variables a real job would use, not their secret values. Keep credentials in the environment; do not paste them into a report.

--smoke only limits task selection. Remove --dry-run and the CLI launches the Harbor jobs, which can make model calls and incur compute and provider costs. Review the generated commands, budget, credentials, and sandbox configuration before launching. Treat a plan as a plan, never as benchmark evidence.

Sources

This walkthrough explains the public CLI at the cited commit. No benchmark was launched and no new performance or cost result is claimed.

Top comments (0)