Free Draft Volume or Paid First Passes: A Debate for Agent SQL Budgets

A nightly analytics job in my lab reproduction turns plain-English questions into SQL, and every draft has to survive a rehearsal database before a human sees it. After three weeks of running that loop, the argument inside the team was never about prompt design. It was about where the budget should go: more drafts on a cheap tier, or fewer, better drafts on an expensive one.

The numbers below are not a product benchmark, because I cannot measure your schema or your tasks. Everything here is a harness you run yourself, and the only claims I make are about the method and the decision rule it produces.

Position A: draft breadth is the bottleneck

The first position says generation is cheap relative to review, so a loop should buy as many independent drafts as it can afford. If a task has a 12% chance of a correct first attempt, ten attempts give you roughly a 72% chance that at least one passes, and that arithmetic is the whole argument. Breadth turns a lucky single shot into an engineering process, but it only works when each attempt costs almost nothing.

This is where free model access and a free server option stop being a discount and become a technical argument. A free draft tier plus a free rehearsal host means the marginal cost of attempt eleven is zero, so your accept@k curve is bounded by wall-clock time rather than by an invoice. MonkeyCode, which its maintainers present as an open-source project, advertises that combination: free model access and a free server option, with a token allowance quoted to me as ten million tokens at the time of writing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat any published quota as a snapshot rather than a contract, because quotas, model versions, and availability change without negotiation. Whatever free capacity you have, pin it in your logs by model identifier and response hash, so a silent model swap appears as a step change in your accept rate instead of as a mystery.

Position B: verification is the bottleneck, so cheap drafts create debt

The second position is less flattering. It says the real constraint is how many changes a human can verify per day, and that cheap drafts simply move cost from the invoice to the reviewer. A draft that fails execution is free to discard, but a draft that parses, executes, and returns the wrong answer is expensive, because only a person notices it.

Under this reading, cost per token is a vanity metric and cost per accepted draft is a distraction. The number that matters is the total cost of one merged change, including the minutes a reviewer spends rejecting plausible SQL. Breadth makes this worse in a specific way: it multiplies the number of semantically wrong queries that pass a structural verifier and reach a human queue.

The counter-evidence for Position A is easy to find in your own logs, and it looks like a distribution of failure reasons. If failures cluster at parse and execution, the verifier absorbs them and breadth is nearly free. If failures cluster at "valid SQL, wrong grain, wrong join, wrong filter", then breadth is multiplying review work rather than reducing it.

The artifact: a harness that measures both positions

The harness below is illustrative and was not executed for this article; run it against your own task set and your own endpoints. It needs Python with the sqlglot and requests packages, plus Docker for the rehearsal database.

Step 1 - freeze a task set with an oracle. Write 25 tasks and give each one a machine-checkable expectation, such as an exact row count or a checksum over the sorted result. Without an oracle you cannot separate execution failures from semantic failures, and the whole decision collapses back into opinion.

Step 2 - provision a disposable rehearsal database. Start a throwaway Postgres container, load schema and fixtures, and run every draft inside BEGIN ... ROLLBACK so one fixture serves all attempts.

docker run --rm -d --name rehearsal -p 55432:5432 -e POSTGRES_PASSWORD=postgres postgres:16-alpine
psql postgresql://postgres:postgres@127.0.0.1:55432/postgres -f schema.sql -f fixtures.sql
export REHEARSAL_DSN=postgresql://postgres:postgres@127.0.0.1:55432/postgres

Step 3 - describe your backends in one config file. A free tier and a paid endpoint usually speak the same chat-completions shape, so a single function can drive both. Replace the identifiers and prices with your own; the values below are placeholders, not product facts.

[
  {"name": "free-hosted", "url": "https://<host>/v1", "key_env": "FREE_KEY",
   "model": "<model-id>", "usd_per_mtok": 0.0},
  {"name": "paid-endpoint", "url": "https://<host>/v1", "key_env": "PAID_KEY",
   "model": "<model-id>", "usd_per_mtok": 0.0}
]

Set usd_per_mtok from your own invoice for the paid endpoint, and keep it at zero only after you have confirmed the free tier's terms and quota for your usage.

Step 4 - draft, verify, and log one JSONL row per attempt. The verifier does three checks in order: parse to a single statement, execute under a short statement timeout, then compare against the task oracle.

import json, os, subprocess, time, sqlglot, requests

def draft(backend, schema_card, prompt, attempt):
    t0 = time.time()
    r = requests.post(
        f"{backend['url']}/chat/completions",
        headers={"Authorization": f"Bearer {os.environ[backend['key_env']]}"},
        json={
            "model": backend["model"],
            "temperature": 0.2 + 0.15 * attempt,
            "messages": [
                {"role": "system", "content": schema_card},
                {"role": "user", "content": prompt},
            ],
        },
        timeout=120,
    )
    r.raise_for_status()
    body = r.json()
    return body["choices"][0]["message"]["content"].strip(), body.get("usage", {}), time.time() - t0

def verify(sql, task, dsn):
    tree = sqlglot.parse(sql, read="postgres")
    if len(tree) != 1:
        return False, "shape", None
    wrapped = "BEGIN; SET LOCAL statement_timeout='2s'; " + sql.rstrip(";") + "; ROLLBACK;"
    p = subprocess.run(
        ["psql", dsn, "-v", "ON_ERROR_STOP=1", "-Atc", wrapped],
        capture_output=True, text=True, timeout=30,
    )
    if p.returncode != 0:
        return False, "exec", p.stderr.strip()[:160]
    rows = [line for line in p.stdout.strip().splitlines() if line]
    ok = len(rows) == task["expected_rows"]
    return ok, "ok" if ok else "oracle", len(rows)

Step 5 - reduce attempts into one comparison table. Group by backend and compute accept rate, cost per accepted draft, and p95 latency, so both positions can be argued with numbers instead of adjectives.

def summarize(rows):
    stats = {}
    for row in rows:
        s = stats.setdefault(row["backend"], {"drafts": 0, "accepted": 0, "usd": 0.0, "lat": []})
        s["drafts"] += 1
        s["accepted"] += int(row["accepted"])
        s["usd"] += row["usd"]
        s["lat"].append(row["latency_s"])
    for s in stats.values():
        s["accept_rate"] = round(s["accepted"] / s["drafts"], 3)
        s["usd_per_accepted"] = round(s["usd"] / max(s["accepted"], 1), 4)
        s["p95_latency_s"] = round(sorted(s["lat"])[int(0.95 * len(s["lat"])) - 1], 1)
    return stats

The decision rule

Run both backends over the same 25 tasks and apply one rule, written as a formula you can defend in a review.

tco_per_merged_change =
    (draft_usd + verify_usd + review_minutes * review_usd_per_min) / accepted

review_minutes comes from your pull-request history rather than from a guess, and in most teams it is the largest term in the equation. Then apply the branches below to pick a configuration instead of an ideology.

Harness signal	Cheapest correct choice	Reason
Free tier's accept rate is within ~30% of paid, failures at parse/exec	Free draft tier, 10-20 attempts	The verifier absorbs failures, so breadth stays cheap
Over a third of failures are "valid SQL, wrong answer"	Paid endpoint, 1-3 attempts, plus an oracle	Review minutes dominate; breadth multiplies debt
Every task needs fresh fixtures	Free server for the rehearsal database	Throwaway rehearsal is the cheapest part of the loop
Schema under data-residency or client-contract limits	Self-hosted only	Any hosted free tier is still a third-party processor

The rule is not "free beats paid" or the reverse. Breadth wins when your verifier is strong and your oracle is per-task; precision wins when your verifier is structural and your oracle is a human in a queue.

Limitations and who should not use this

Skip the breadth-first loop if you cannot write a per-task oracle, because you will be multiplying exactly the failures you are unable to detect. Skip any hosted free tier if schema cards or sample rows fall under a data-processing agreement that does not cover that provider, and check the terms before the first request rather than after the first audit.

Do not plan a budget around a published quota, and do not assume a free tier is permanent. Re-run the harness after any provider change, because your accept-rate history is the only early warning you will get.

This workflow is also a poor fit for one-shot schema migrations. There, review is the only gate you have, and ten cheap drafts simply hand a reviewer ten ways to be wrong.

If you want to test the breadth-first branch without paying per draft, the free tier is the cheapest way to get the harness above running - measure it with the same oracle and the same decision rule before you commit to it.