Prototype Multi-Agent Ideas on a Zero Budget: An Evaluation-First Workflow

Scrolling DEV this week, half the posts in my feed are about agents: orchestration frameworks, A2A protocols, sub-agent metrics, local AI rigs. The pattern I keep hitting when I try these ideas myself is the same one every time: I burn through paid API quota (or weekend GPU time) just to discover that my agent design was wrong at step two.

So I flipped the order. Now I write the evaluation harness first, run it against free model access on a free hosted environment, and only spend money after the harness tells me the design is worth spending on. This post is that workflow, with a runnable harness you can copy.

The problem: agent prototypes fail expensively

A single-agent script is cheap to iterate on. The moment you add a planner, a worker, and a critic, every design experiment costs tokens across all three roles, plus a place to run the loop. If you discover — as I usually do — that the planner's prompt is the actual bottleneck, you've paid to learn something a free tier could have told you.

The fix is to separate two questions that are easy to conflate:

1. Is the design sound? (task decomposition, routing, termination conditions)
2. Is it fast/cheap/smart enough for production? (latency, cost per task, frontier-model quality)

Question 1 is structural. A weaker free model can answer it. Question 2 genuinely needs your production stack — and should be deferred until question 1 passes.

The artifact: an evaluation harness that runs before any infrastructure

Here's the harness I use. It's deliberately model-agnostic: it scores a pipeline's structure (did the right sub-tasks get produced? did the loop terminate? did the critic reject bad output?) rather than prose quality, so it stays meaningful even on small free models.

# eval_harness.py — structural evaluation for a multi-agent pipeline
# Runs against any OpenAI-compatible endpoint (free tier or paid).
# Label: tested locally against a mock client; swap in your real client.

import json, time
from dataclasses import dataclass, field

@dataclass
class TaskCase:
    name: str
    goal: str
    required_subtasks: list[str]   # keywords the planner MUST produce
    forbidden_subtasks: list[str]  # signs of scope creep
    max_steps: int = 8             # termination check

@dataclass
class Result:
    name: str
    passed: bool
    failures: list[str] = field(default_factory=list)
    wall_seconds: float = 0.0

CASES = [
    TaskCase(
        name="summarize_and_email",
        goal="Summarize this report and draft an email to the team",
        required_subtasks=["summarize", "draft"],
        forbidden_subtasks=["send"],  # agent must not act, only draft
    ),
    TaskCase(
        name="ambiguous_goal",
        goal="Make the dashboard better",
        required_subtasks=["clarify"],  # good planners ask before acting
        forbidden_subtasks=[],
        max_steps=3,  # should terminate fast by asking
    ),
]

def run_case(client, case: TaskCase) -> Result:
    start = time.time()
    failures = []

    # planner step
    plan = client.chat(
        system="You are a planner. List sub-tasks, one per line. "
               "If the goal is ambiguous, your first line must be 'clarify: <question>'.",
        user=case.goal,
    )
    steps = [s.strip().lower() for s in plan.splitlines() if s.strip()]

    # structural checks
    if len(steps) > case.max_steps:
        failures.append(f"plan too long: {len(steps)} steps > {case.max_steps}")
    for req in case.required_subtasks:
        if not any(req in s for s in steps):
            failures.append(f"missing required sub-task: '{req}'")
    for bad in case.forbidden_subtasks:
        if any(bad in s for s in steps):
            failures.append(f"forbidden sub-task present: '{bad}'")

    return Result(case.name, not failures, failures, time.time() - start)

def main(client):
    results = [run_case(client, c) for c in CASES]
    for r in results:
        status = "PASS" if r.passed else "FAIL"
        print(f"[{status}] {r.name} ({r.wall_seconds:.1f}s)")
        for f in r.failures:
            print(f"    - {f}")
    passed = sum(r.passed for r in results)
    print(f"\n{passed}/{len(results)} structural checks passed")
    return 0 if passed == len(results) else 1

Two things worth noticing:

• The ambiguous_goal case is the one that catches the most real design bugs. Many planner prompts will happily invent sub-tasks for "make the dashboard better" instead of asking a clarifying question. That failure costs nothing to find on a free model.
• wall_seconds is recorded but intentionally not scored. Latency on a free shared environment tells you nothing about production latency. Scoring it would be measuring the wrong thing.

Where the free sandbox fits

You need two things to run this loop: model inference, and somewhere to execute the harness. I use MonkeyCode for both — its free model access covers the inference side, and its free server option means the harness runs somewhere other than my laptop, which matters when a teammate needs to re-run the same eval against the same setup.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I want to be precise about what I am not claiming: I haven't verified which models are available, what the quotas are, or how long the free options will exist, so treat all of that as subject to change and check the current terms before building anything on top of it. The workflow below does not depend on any of those specifics — that's the point. If the free option disappears, you point client at a different endpoint and the harness still works.

The decision table: when to graduate to paid infrastructure

Signal from the harness	Verdict	Next step
Structural checks fail on free models	Design problem, not a model problem	Fix prompts/decomposition; stay on free tier
Structural checks pass, output quality is poor	Possibly a model-capability ceiling	Re-run the same harness on one paid model to confirm
Structural checks pass on a paid model too	Design is sound	Now benchmark latency/cost on production infra
Pass rate is flaky across runs	Non-deterministic decomposition	Add constraints (schemas, few-shot examples) before spending anything

The row that saves the most money is the second one: you re-run the identical harness on a single paid call instead of rewriting your prototype around a paid model and hoping.

Limitations, honestly

• Structural checks are not quality checks. A plan can contain all the right keywords and still be a bad plan. The harness filters out broken designs; it doesn't certify good ones.
• Free shared environments are noisy. Don't time anything there, don't compare model A vs model B on latency there, and don't assume availability for CI.
• Free tiers end. Anything you wire into a nightly job should be portable. Keep the client behind an interface (the harness above assumes an OpenAI-compatible shape for exactly this reason).

Who should not use this approach

• If your task is safety-critical or handles private user data, don't send it to any free hosted environment — evaluate locally with an open model instead.
• If you're benchmarking production latency or cost per task, a free tier will give you numbers that are wrong in both directions. Skip to paid infra with a small sample.
• If your agent's core risk is tool integration (auth, side effects, rate limits) rather than decomposition, a structural harness doesn't test your actual risk. Build a tool-call mock harness instead.

Closing

The cheapest token is the one you don't spend discovering a broken design. Write the structural evaluation first, run it somewhere free, and let the harness — not your optimism — decide when the idea has earned a real budget. If you want to try the loop this week, the script above plus any free model endpoint is genuinely all you need; MonkeyCode's free tier happens to be the combination I'm using, but the harness outlives any single provider.