Benchmarking AI SQL Fixes on a Free Server: A Structured Debate on What You Can Actually Trust

Last Tuesday, our eval harness reported that an AI SQL-fixing model cut its average latency from 1,900 milliseconds to 900 milliseconds overnight, and someone started drafting a migration ticket before anyone inspected the run conditions. The next morning, the identical harness returned a 2.1-second average on the same tasks, which transformed a celebration into a measurement dispute that consumed the afternoon. The real culprit was neither the model nor the SQL corpus; it was shared capacity on a free AI server, combined with zero warm-up discipline and a benchmark that mixed correctness with timing. That incident is why this article is a structured debate rather than a tutorial, because the honest answer depends on which question you ask the benchmark.

This article presents two credible positions about running SQL-fix evals on free infrastructure, a decision rule that reconciles them, and a runnable Python harness that separates functional correctness from latency measurement. As the concrete example, I used MonkeyCode's free model access and its free server option, because those two availability claims are easy to verify and the entire debate is about zero-cost infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Position A: Free Servers Make Evals Frequent, and Frequency Finds Regressions

The first position argues that a free server with free model access is the best eval environment precisely because it removes the economic excuse for skipping a run. When every execution costs nothing, a team can run the harness nightly, after every prompt change, and even after every documentation edit that touches the system prompt. A regression that appears only on day three of nightly runs would never surface in a weekly paid run, and the expected time-to-detection shrinks as run frequency rises.

The evidence from our own incident supports this arithmetic. A prompt-formatting change on day three produced a measurable drop in plan-shape accuracy, and only the nightly schedule caught it before the change reached production. The latency numbers were noise, but the correctness signal was real, and that signal justifies the free infrastructure all by itself.

The strongest form of this position is a cost argument: a free model tier plus a free server converts evaluation from a budgeted event into a default habit. Teams that evaluate daily observe regressions in hours or days instead of weeks, and they build an institutional memory of failure modes that paid, infrequent runs cannot provide.

Position B: Shared Capacity Turns Timing Data Into Statistical Noise

The second position concedes the correctness argument but insists that any latency or throughput claim from a free shared server is untrustworthy. Cold starts, throttling, noisy neighbors, and queueing all create variance that swamps the differences a benchmark is trying to detect. Our three runs tell the story: 1.9 seconds, 0.9 seconds, and 2.1 seconds on identical tasks, which means a 50 percent latency delta between two models is not a signal at all.

The practical consequence is that free-server benchmarks can only answer functional questions like "does the generated SQL return the right result" or "does the plan shape match the expected pattern." They cannot answer operational questions like "is model A faster than model B for our workload," and anyone who publishes such numbers without a controlled environment is publishing noise with a confidence interval attached.

This position is not a rejection of free infrastructure; it is a boundary condition. Use the free server for gates that catch regressions, and reserve timing conclusions for a dedicated environment with warm-up phases, interleaved trials, and a pre-registered analysis.

The Decision Rule: Split Correctness From Timing, Then Trust Only What Passes

Both positions are correct within their domains, and the way out is a three-step decision rule. Follow these steps in order, and the benchmark will tell you what you need without lying.

1. Write your question down first. The eval must answer one of two questions: "did the fix regress functionally" or "is one approach faster." A benchmark that tries to answer both at once will answer neither honestly.
2. Gate on correctness before you look at timing. Accept a result only when the generated SQL executes without error, returns the expected rows, and matches the expected plan shape. Everything else is a rejected sample.
3. If timing matters, measure it as a relative signal. Run interleaved trials of both candidates, discard the first call after a cold start, and report median plus interquartile range rather than averages. Label every number as "relative signal on a shared free server," never as an absolute performance guarantee.

The harness below implements this rule in about sixty lines, and you can drop it into any repository that already has an API client.

# eval_split.py - separates functional correctness from timing noise
import statistics
import time
from dataclasses import dataclass


@dataclass
class Task:
    id: str
    schema: str
    question: str
    expected_plan_shape: str


@dataclass
class Result:
    task_id: str
    correct: bool
    latency_ms: float
    cold: bool


def run_single(client, task, cold):
    if cold:
        client.complete(task.schema, 'SELECT 1')  # warm-up call
    start = time.perf_counter()
    response = client.complete(task.schema, task.question)
    latency_ms = (time.perf_counter() - start) * 1000
    correct = response.get('plan_shape') == task.expected_plan_shape
    return Result(task.id, correct, latency_ms, cold)


def evaluate(client, tasks, warmup=1):
    results = []
    for task in tasks:
        for _ in range(warmup):
            results.append(run_single(client, task, cold=True))
        for _ in range(max(3 - warmup, 1)):  # interleaved warm trials
            results.append(run_single(client, task, cold=False))
    return results


def report(results):
    accuracy = sum(r.correct for r in results) / len(results)
    timed = [r for r in results if r.correct]
    for cold in (True, False):
        group = sorted(r.latency_ms for r in timed if r.cold is cold)
        if group:
            median = statistics.median(group)
            lo = group[len(group) // 4]
            hi = group[3 * len(group) // 4]
            print(f'cold={cold} n={len(group)} median={median:.0f}ms IQR={lo:.0f}-{hi:.0f}ms')
    print(f'accuracy={accuracy:.2%}')

The script deliberately discards timing data for incorrect results because a fast wrong answer is still wrong. The warm-up call isolates the cold-start penalty, and the interleaved trial design makes a noisy neighbor affect both candidates equally.

When This Approach Should Not Be Used

Teams that publish vendor benchmarks, capacity planners, and anyone whose SLO depends on p95 latency should not trust timing numbers from a free shared server, no matter how careful the warm-up phase is. The harness also assumes your client returns a plan_shape field, so adapt it if your endpoint returns raw SQL or free text instead. If your decision changes a production budget, run the final validation on dedicated infrastructure and treat the free server as a screening stage only.

If you want to reproduce these conditions with your own SQL corpus, the script is self-contained, and MonkeyCode's free server is one convenient place to point it at. Run it for three nights, keep the correctness column, and treat the timing column as a conversation starter rather than a conclusion.