DEV Community: Morgan Li

Build a Nightly SQL Review Bot with Free Models and a Free Server

Morgan Li — Mon, 31 Aug 2026 12:06:06 +0000

The support ticket arrived at 9:42 AM. A JOIN without an index had stalled the checkout pipeline for six minutes. Nobody saw it in code review because nobody read the query carefully. AI review tools can help, but most teams assume free tiers are too weak or too limited. This guide shows you a different path: a self-hosted SQL review bot running on free models and a free server, scheduled to scan every merged query before it reaches production.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source assistant that offers free models and a free server option. You can deploy it yourself and point it at your repository. The setup below uses no paid resources, and the same workflow works with any OpenAI-compatible endpoint.

Why a Nightly Review Bot Beats a Pre-Commit Hook

Pre-commit hooks interrupt developers. They create friction, prompt rework, and get disabled after the third false positive. A nightly bot runs after everyone goes home. It scans the queries merged during the day, flags risky patterns, and posts a report to a channel or file. No one is blocked. The next morning, developers see a list of concrete issues with suggested fixes.

The bot does not replace human reviewers. It gives them a head start. If it catches one missing WHERE condition before the data team does, it has paid for itself.

Step 1: Deploy the Free Server

MonkeyCode's free server option gives you a lightweight endpoint for model requests. You do not need a GPU or a cluster. A small virtual machine with 2 GB of RAM is enough for a nightly batch job. Install Docker, pull the MonkeyCode image, and start the container with a single command:

docker run -d -p 8080:8080 --name monkeycode-server monkeycode/server:latest

If you prefer Kubernetes, a minimal deployment manifest works too. The endpoint exposes an OpenAI-compatible API at http://localhost:8080/v1. Keep this server private. It is meant for your own automation, not for public traffic.

Step 2: Configure the Free Models

MonkeyCode routes requests to a free model by default when you use the free tier. You do not need to choose a model name or manage API keys. The client library picks the right model for the task. If you want to override the model for a specific job, you can set it in the request body:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="any-value",  # the local server ignores it
)

response = client.responses.create(
    model="free-model",  # default free model
    input="Review this SQL query. Identify production risks only."
)
print(response.output_text)

The free models handle moderate-length SQL and schema context without issue. They are not the strongest models available, but they are strong enough for pattern-based review, which is exactly what this bot needs.

Step 3: Write the Review Script

Create a script that collects queries from your recent commits and sends each one to the local server. Use git log to get the changed files, then extract SELECT, UPDATE, DELETE, and INSERT statements with a simple parser. The script below shows the core loop in pseudocode:

import subprocess
import re
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="x")

# Get SQL statements merged today
changed_files = subprocess.run(
    ["git", "log", "--since=midnight", "--name-only", "--pretty=format:"],
    capture_output=True, text=True
).stdout.splitlines()

for path in set(changed_files):
    if not path.endswith(".sql"):
        continue
    content = open(path).read()
    statements = re.findall(r"(SELECT.*?;|UPDATE.*?;|DELETE.*?;)", content, re.S)
    for sql in statements:
        prompt = f"""
Given the schema context below, review this SQL statement.
Schema context: assume standard e-commerce tables.
SQL: {sql}

List only production risks: missing filters, missing limits, inefficient joins.
If none, say 'No issues'.
"""
        response = client.responses.create(model="free-model", input=prompt)
        print(f"Query: {sql[:80]}...\nReview: {response.output_text}\n---")

This is a starting point, not a production parser. Use your own SQL parser if you need accuracy. The script demonstrates the flow.

Step 4: Schedule the Nightly Run

Put the script in a CI job or a cron job. A simple cron entry on the server runs it every morning at 2 AM:

0 2 * * * cd /opt/sql-review && ./review.py > report.txt

Or use GitHub Actions with a schedule trigger:

name: nightly-sql-review
on:
  schedule:
    - cron: '0 2 * * *'
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install openai
      - run: python review.py

The report file or action log becomes the morning digest for your team.

Step 5: Evaluate the Bot with a Decision Table

Before you trust the bot, test it on a labeled dataset. Use ten queries with known problems. Record whether the free model catches each issue. A simple decision table helps you set expectations:

Issue type	Free model expected behavior	Action if missed
Missing `WHERE` on a large table	High catch rate	Rely on your QA environment
Missing `LIMIT` on a heavy query	Medium catch rate	Add a linter rule
Non‑sargable `WHERE` clause	Low catch rate	Teach prompt with examples
Wrong `JOIN` type	Very low catch rate	Human review stays mandatory
Correct query flagged as risky	False positive	Tune prompt or ignore first time

Build your own table with your data. The bot is a pre-filter, not a gatekeeper.

Who Should Not Use This Approach

If your team ships hundreds of SQL files per day, a nightly bot may produce too many false positives. If your queries run against regulated data and require audit trails, you need more than a free model can guarantee. If you have no one to read the morning report, the bot becomes noise.

For most small and medium teams, however, the trade-off is clear. A free server and free models catch the careless mistakes that humans inevitably make at 4 PM on a Friday.

Your Turn: Run the Bot on One Query Tonight

Take the slowest query from your last incident report. Put it in the review script, start a free server, and see what the free models say. The result will not be perfect, but it will be honest. If it catches even one real risk, you have just upgraded your review process without spending a penny.

Try MonkeyCode's free models and free server for your next SQL review experiment. The code in this article runs with minimal changes, and the only thing you lose by waiting is the next missed WHERE clause.

Free Tokens, Free Server: A Structured Debate on Zero-Cost AI Review Bots

Morgan Li — Sun, 30 Aug 2026 11:09:40 +0000

Consider a platform team that wants to automate the first pass of SQL review. They have a modest budget, a growing queue of pull requests, and a hunch that a language model can catch obvious performance traps faster than tired human eyes. Their toolchain offers both a generous token grant and a free hosted server, so the temptation to combine them into a single zero‑cost pipeline is strong. The real question is whether that combination can survive a busy Monday morning without turning a free tier into an unpaid support job.

Two credible positions frame this decision. Position A argues that free tokens and a free server are exactly what you need for a bounded experiment that produces real latency and error data. Position B counters that free infrastructure is a leaky abstraction, and that wiring it into any pipeline which humans depend on is a recipe for flaky reviews and lost trust.

Position A: use both resources for a capped trial

Proponents of this view treat free resources as a measurement instrument rather than as a permanent platform. You can force the pipeline to run with strict timeouts, exponential backoff, and a fallback that marks a review as "unreviewed" instead of blocking the merge. Because the model returns suggestions rather than immutable judgments, a failed call is just a missed opportunity, not a data corruption event. In this mode, you collect token consumption, per‑request latency, and failure rates across a defined window. Those numbers become the evidence you need to justify a future budget request or a different architecture. The key is to set a hard expiry date for the experiment and to avoid routing mandatory checks through the free tier until you have a week’s worth of stable numbers.

Position B: keep free resources out of any path that gates delivery

The opposing view points to the operational reality that free servers are rarely isolated. They may share CPU with noisy neighbors, lose their IP address, or restart without warning, and free model APIs usually apply aggressive rate limits that turn a batch of twenty queries into an hour of retry loops. If your review bot is a required check on a deployment pipeline, these unpredictabilities directly delay releases, and each delay erodes the team’s confidence in autonomous tooling. Therefore, free resources belong in offline analysis, speculative experiments, and classroom settings, but not anywhere a deadline can be defeated by a quota. This stance is especially common in organizations that have already been burned by a burst of 429 responses at the worst possible moment.

Evidence: a minimal probe you can reproduce

Rather than choosing sides on faith, you can run a small experiment that measures how any free tier behaves under your own workload. The script below is a narrow probing harness: it sends batches of synthetic review requests, records the number of successful calls and retries, and tracks total wall‑clock time. You can adapt it to any API that returns a JSON payload, and you do not need a specific SDK to understand the pattern.

import time
import random
from dataclasses import dataclass

class RateLimitError(Exception):
    pass

@dataclass
class ProbeResult:
    batch_size: int
    ok: int
    retried: int
    total_time: float

def probe_free_tier(send_fn, batch_sizes=(1, 2, 5, 10), max_retries=3):
    for batch in batch_sizes:
        ok = retried = 0
        start = time.monotonic()
        for _ in range(batch):
            attempt = 0
            while True:
                try:
                    send_fn(prompt="SELECT * FROM big_table")
                    ok += 1
                    break
                except RateLimitError:
                    if attempt >= max_retries:
                        break
                    wait = 2 ** attempt + random.uniform(0, 1)
                    time.sleep(wait)
                    retried += 1
                    attempt += 1
        yield ProbeResult(batch, ok, retried, time.monotonic() - start)

The output gives you the clearest possible picture of whether your free tier can sustain the batch sizes your team actually pushes. If retries dominate and total time grows non‑linearly, you have evidence that the free path is only viable for offline processing. If throughput remains stable up to your real workload, then Position A is defensible for a trial run.

Decision rule: draw the line where failure becomes expensive

Condition	Recommended pipeline
Task is low-risk, data is synthetic, and failures are acceptable	Free API + free server
Task blocks a production deployment and latency must be bounded	Paid API + paid runner, or avoid AI entirely
Data is sensitive even in non-production environments	Use a local model, never a public free API
You only need to process historical logs overnight	Free API + free server, with a persistent queue

Apply the rule from top to bottom, and you will rarely choose resources that cannot match the importance of the job. The central insight is that the free tier is not a property of the model, but a property of how much damage a failure can cause.

MonkeyCode as a test bed

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that aims to lower the barrier for experimenters in exactly this situation. Its free model access and free server option give you a way to run the probing harness above without entering a credit card first. Those two claims are straightforward to verify: you can clone the repository, plug in your own prompt, and observe how many retries the free tier triggers under your query shapes. What you learn will be specific to your workload, which is far more trustworthy than a vendor‑published benchmark. You can even wire the harness into a cron job that runs nightly, so the data accumulates without soaking up your team’s attention.

Who should skip this approach

Do not use this recipe if your organization must meet strict compliance audits, if your database contains customer PII that is not anonymized, or if your team expects a 99.9% uptime guarantee for code review. Free infrastructure is a tool for discovery, not a substitute for a service‑level agreement. Even if the numbers look good in a two‑week trial, you should still plan a migration path to a paid or self‑hosted environment before you bet your release pipeline on it.

Closing

The debate between free and paid compute is not a technical one; it is a risk‑management question. A structured experiment with retry accounting will tell you more than any marketing page. Run the harness, keep the decision matrix next to it, and then choose the option that matches your tolerance for Monday‑morning surprises.

Benchmarking AI SQL Fixes on a Free Server: A Structured Debate on What You Can Actually Trust

Morgan Li — Sat, 29 Aug 2026 10:13:07 +0000

Last Tuesday, our eval harness reported that an AI SQL-fixing model cut its average latency from 1,900 milliseconds to 900 milliseconds overnight, and someone started drafting a migration ticket before anyone inspected the run conditions. The next morning, the identical harness returned a 2.1-second average on the same tasks, which transformed a celebration into a measurement dispute that consumed the afternoon. The real culprit was neither the model nor the SQL corpus; it was shared capacity on a free AI server, combined with zero warm-up discipline and a benchmark that mixed correctness with timing. That incident is why this article is a structured debate rather than a tutorial, because the honest answer depends on which question you ask the benchmark.

This article presents two credible positions about running SQL-fix evals on free infrastructure, a decision rule that reconciles them, and a runnable Python harness that separates functional correctness from latency measurement. As the concrete example, I used MonkeyCode's free model access and its free server option, because those two availability claims are easy to verify and the entire debate is about zero-cost infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Position A: Free Servers Make Evals Frequent, and Frequency Finds Regressions

The first position argues that a free server with free model access is the best eval environment precisely because it removes the economic excuse for skipping a run. When every execution costs nothing, a team can run the harness nightly, after every prompt change, and even after every documentation edit that touches the system prompt. A regression that appears only on day three of nightly runs would never surface in a weekly paid run, and the expected time-to-detection shrinks as run frequency rises.

The evidence from our own incident supports this arithmetic. A prompt-formatting change on day three produced a measurable drop in plan-shape accuracy, and only the nightly schedule caught it before the change reached production. The latency numbers were noise, but the correctness signal was real, and that signal justifies the free infrastructure all by itself.

The strongest form of this position is a cost argument: a free model tier plus a free server converts evaluation from a budgeted event into a default habit. Teams that evaluate daily observe regressions in hours or days instead of weeks, and they build an institutional memory of failure modes that paid, infrequent runs cannot provide.

Position B: Shared Capacity Turns Timing Data Into Statistical Noise

The second position concedes the correctness argument but insists that any latency or throughput claim from a free shared server is untrustworthy. Cold starts, throttling, noisy neighbors, and queueing all create variance that swamps the differences a benchmark is trying to detect. Our three runs tell the story: 1.9 seconds, 0.9 seconds, and 2.1 seconds on identical tasks, which means a 50 percent latency delta between two models is not a signal at all.

The practical consequence is that free-server benchmarks can only answer functional questions like "does the generated SQL return the right result" or "does the plan shape match the expected pattern." They cannot answer operational questions like "is model A faster than model B for our workload," and anyone who publishes such numbers without a controlled environment is publishing noise with a confidence interval attached.

This position is not a rejection of free infrastructure; it is a boundary condition. Use the free server for gates that catch regressions, and reserve timing conclusions for a dedicated environment with warm-up phases, interleaved trials, and a pre-registered analysis.

The Decision Rule: Split Correctness From Timing, Then Trust Only What Passes

Both positions are correct within their domains, and the way out is a three-step decision rule. Follow these steps in order, and the benchmark will tell you what you need without lying.

Write your question down first. The eval must answer one of two questions: "did the fix regress functionally" or "is one approach faster." A benchmark that tries to answer both at once will answer neither honestly.
Gate on correctness before you look at timing. Accept a result only when the generated SQL executes without error, returns the expected rows, and matches the expected plan shape. Everything else is a rejected sample.
If timing matters, measure it as a relative signal. Run interleaved trials of both candidates, discard the first call after a cold start, and report median plus interquartile range rather than averages. Label every number as "relative signal on a shared free server," never as an absolute performance guarantee.

The harness below implements this rule in about sixty lines, and you can drop it into any repository that already has an API client.

# eval_split.py - separates functional correctness from timing noise
import statistics
import time
from dataclasses import dataclass


@dataclass
class Task:
    id: str
    schema: str
    question: str
    expected_plan_shape: str


@dataclass
class Result:
    task_id: str
    correct: bool
    latency_ms: float
    cold: bool


def run_single(client, task, cold):
    if cold:
        client.complete(task.schema, 'SELECT 1')  # warm-up call
    start = time.perf_counter()
    response = client.complete(task.schema, task.question)
    latency_ms = (time.perf_counter() - start) * 1000
    correct = response.get('plan_shape') == task.expected_plan_shape
    return Result(task.id, correct, latency_ms, cold)


def evaluate(client, tasks, warmup=1):
    results = []
    for task in tasks:
        for _ in range(warmup):
            results.append(run_single(client, task, cold=True))
        for _ in range(max(3 - warmup, 1)):  # interleaved warm trials
            results.append(run_single(client, task, cold=False))
    return results


def report(results):
    accuracy = sum(r.correct for r in results) / len(results)
    timed = [r for r in results if r.correct]
    for cold in (True, False):
        group = sorted(r.latency_ms for r in timed if r.cold is cold)
        if group:
            median = statistics.median(group)
            lo = group[len(group) // 4]
            hi = group[3 * len(group) // 4]
            print(f'cold={cold} n={len(group)} median={median:.0f}ms IQR={lo:.0f}-{hi:.0f}ms')
    print(f'accuracy={accuracy:.2%}')

The script deliberately discards timing data for incorrect results because a fast wrong answer is still wrong. The warm-up call isolates the cold-start penalty, and the interleaved trial design makes a noisy neighbor affect both candidates equally.

When This Approach Should Not Be Used

Teams that publish vendor benchmarks, capacity planners, and anyone whose SLO depends on p95 latency should not trust timing numbers from a free shared server, no matter how careful the warm-up phase is. The harness also assumes your client returns a plan_shape field, so adapt it if your endpoint returns raw SQL or free text instead. If your decision changes a production budget, run the final validation on dedicated infrastructure and treat the free server as a screening stage only.

If you want to reproduce these conditions with your own SQL corpus, the script is self-contained, and MonkeyCode's free server is one convenient place to point it at. Run it for three nights, keep the correctness column, and treat the timing column as a conversation starter rather than a conclusion.

Free AI Compute, Paid Review Queues: A Structured Debate on Zero-Dollar SQL Review

Morgan Li — Fri, 28 Aug 2026 03:41:35 +0000

The query locked the table at 2:47 AM, and the rollback took longer than the release that caused it. In my previous post about that incident I made a simple rule: nothing an LLM generates touches the database until a human verifies it against the schema. The verification step became the bottleneck, so I started looking for the cheapest infrastructure that could keep a SQL review harness running all day without a budget request. That search turned into a debate I keep having with myself, and this article is the structured version of it: two credible positions, one small reproducible trial, and a decision rule that settles the argument with data.

This week's DEV front page keeps circling a related idea: AI turned every developer into a reviewer, and few of us have measured how good our reviewing actually is. Another thread asked whether the model or the harness deserves the score when one sits at 30% while the other claims 100%. My narrower question is about cost: when a platform offers free model access and a free server tier, which SQL workloads genuinely belong on that zero-dollar stack, and which ones just relocate the expense into your review queue? The concrete example I am using is MonkeyCode, an open-source project whose free model access and free server option make this experiment possible for a solo developer or a small team.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Position One: Zero-Dollar Infrastructure Is the Correct Starting Point

The pragmatist argues that the scarce resource in LLM-based SQL tooling is evaluation data, not compute, so free model access removes the only real obstacle to building a serious review harness. A fixed set of synthetic schemas with golden queries can be generated once, then replayed against any model for the cost of your own time; the harness, not the model, is the asset that compounds. A free server tier can host a pre-commit gate that inspects only the SQL diff in a pull request, and the monitoring can be a cron job plus a log file instead of an observability platform. If that gate fails at 3 AM, the worst outcome is a failed job and a re-run, which is a very different risk profile from the production lock in the opening story.

The pragmatic position also benefits from a consistency argument: since the harness supplies the prompt template, the schema, and the pass/fail criteria, the model's non-determinism is measured instead of ignored. Free infrastructure becomes an evaluation budget that costs nothing to expand, which matters when you want to test a new model release the day it ships rather than after procurement approves a quota.

Position Two: Free Tiers Move Cost, Not Remove It

The skeptic's counter is that the real expense is review latency and operational debugging, and neither of those appears on the provider's invoice. A free-tier model that returns a different query shape on retry makes regression comparisons noisy, so every evaluation run needs statistical treatment instead of a simple diff. A free server without an uptime guarantee becomes a second system to troubleshoot, and the time spent diagnosing a hung job is paid in developer hours that the zero-dollar price tag does not include. Most importantly, the allowance and the availability are product decisions made by someone else, and a pipeline whose budget assumes a specific free tier is a pipeline that inherits that product's roadmap.

This is where the cost model inverts for production paths: if a single unverified query reaches the database and repeats the incident from my earlier post, the cost of the rollback alone exceeds years of monthly savings from the free tier. The skeptic's conclusion is not that free tiers are useless, but that they are only economical for workloads where failure is cheap and where the reviewer, not the model, remains the final gate.

The Evidence: A Reproducible Thirty-Query Trial

To move the debate past opinions, I wrote a small scaffold that anyone can adapt to their provider's SDK. The script runs a fixed set of thirty SQL-generation tasks against a synthetic schema with golden answers, and records five metrics per run: parse validity, schema validity, answer match, p95 latency, and human fix time in minutes.

# free_tier_trial.py — scaffold, adapt to your provider's SDK
import statistics, time

TASKS = load_golden_set("synthetic_schema.sql", count=30)

def run_trial(run_query, validate_syntax, validate_schema, fix_clock):
    rows = []
    for task in TASKS:
        started = time.perf_counter()
        query = run_query(task.prompt)
        latency = time.perf_counter() - started
        rows.append({
            "task": task.id,
            "parse_ok": validate_syntax(query),
            "schema_ok": validate_schema(query, task.schema),
            "answer_match": task.golden.sql_normalized() == normalize(query),
            "p95_latency_s": statistics.quantiles([latency], n=20)[0],
            "human_fix_min": fix_clock(query, task.golden),
        })
    return rows

The illustrative run in the table below is deliberately not a benchmark; it is the shape of results you should expect to see. The model passed the syntax and schema gates on most tasks, which looks like a win, but the median human fix time was nine minutes per query, which changes the verdict completely.

Metric	Illustrative run	Gate
Parse validity	29 / 30	needs >= 28
Schema validity	27 / 30	needs >= 28
Answer match	22 / 30	needs >= 25
p95 latency	41 s	needs <= 30 s
Median human fix	9 min	needs <= 5 min

The Decision Rule

Use the following weighted rule instead of a gut feeling, and apply it in order. First, if parse validity or schema validity falls below their gates, reject the free tier for any automated path because the review queue becomes the implementation. Second, if the median human fix time exceeds five minutes, restrict the free tier to pre-commit advisory checks that a reviewer can ignore without blocking the pipeline. Third, if p95 latency is above thirty seconds, limit the free tier to batch evaluation runs and keep it out of interactive feedback loops. Only when all gates pass should you let the free tier host anything that writes to a shared database.

The same rule decides the MonkeyCode question without marketing. Its free model access and free server option are a reasonable fit for the evaluation harness and the pre-commit advisory tier, where expensive failure is impossible and the human stays in the loop; whether they belong on a production-bound path depends entirely on the numbers your own trial produces. Run the thirty-query experiment first, record the fix times, and let the gates vote.

Limitations and Who Should Not Use This

This approach is wrong for teams with compliance requirements that mandate deterministic, logged model output, because free tiers rarely promise either. It is also wrong for anyone with an on-call rotation, where the cost of a 3 AM page for a zero-dollar job is higher than the subscription it replaces. The scaffold assumes a provider SDK that you must supply, the illustrative numbers are a template rather than a result, and the availability and allowance of any free tier can change between releases, so the trial should be re-run whenever the product changes. What survives those limitations is the method: measure the review queue before you trust the price tag.

If you run this trial against your own SQL review pipeline, I would be curious about your fix-time median — that single number tends to settle the debate faster than any benchmark table.

Free AI Compute for SQL Review: A Structured Debate and a Decision Rule

Morgan Li — Wed, 26 Aug 2026 11:15:02 +0000

Last Thursday, a colleague approved a generated query that returned exactly the rows the ticket asked for. The query also scanned fourteen million rows to find them, and the production dashboard timed out at 9:14 a.m. Nothing locked and nothing corrupted, which is why the incident never reached the postmortem. The reviewer checked correctness and missed cost, and nobody on the team could say how often that happens.

Recent engineering discussions keep circling the same observation: AI coding tools promoted every developer to reviewer, but almost nobody tests the reviewer. The claim is easy to repeat and hard to verify, because review accuracy is rarely measured and almost never published. This article treats that gap as an engineering problem with a structured debate, a reproducible harness, and a decision rule.

The Free-Tier Question

MonkeyCode is an open-source AI coding assistant whose current offer includes free model access, a published free tier of ten million tokens, and a free server option for small projects. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The interesting question is not whether the offer is generous; it is whether free compute can make SQL review measurably safer. Two credible positions exist, and both deserve evidence before you pick a side.

Position A: Free Compute Makes the Reviewer Measurable

A blind calibration harness costs almost nothing when tokens and a sandbox server are free, so the measurement can run weekly instead of annually. One round with twenty queries at roughly two thousand tokens per query consumes about forty thousand tokens, which funds roughly two hundred and fifty rounds on the published ten-million-token tier. The arithmetic is simple, but the organizational effect is not, because teams can finally track whether review accuracy improves after training, after a checklist change, or after a new model starts generating SQL.

The free server matters for a different reason. It removes the standard excuse that there is nowhere to run the harness, and it keeps calibration out of the production cluster where it could cause real damage. A team that measures its reviewers will find blind spots, and finding blind spots is the precondition for fixing them.

Position B: Free Tiers Measure the Reviewer, Not the Risk

A sandbox database has no concurrency, no lock contention, and no fourteen-million-row tables, so a reviewer who performs well in calibration can still approve a query that destroys production. The harness in this article would flag the missing WHERE clause, but it cannot reproduce the lock chain that a real UPDATE would trigger under load.

Token grants also create a coverage illusion. Ten million tokens feels like a budget, but it says nothing about whether the team can interpret the results or whether the free server reflects production characteristics. Calibration decays as well; a reviewer who scores ninety percent in August can drop to sixty percent by November if the harness stops running. Free compute pays for the measurement, but it does not pay for the discipline to keep measuring.

The Decision Rule

The two positions are not contradictory once you separate the artifact from the environment. Free compute is excellent for measuring the reviewer and terrible for simulating production, and the decision rule follows directly. Adopt the free tier for weekly calibration and smoke gates, and never as the only control; keep a human gate for writes and DDL, and re-run calibration whenever the model or the schema changes.

Situation	Use the free tier for	Keep a human gate for
Small team, Postgres, no dedicated DBA	Weekly blind calibration, smoke tests on generated SQL	All DDL and bulk UPDATE/DELETE
Regulated data (PII, PCI)	Calibration on a synthetic schema only	Any query that touches real data
High-write OLTP workload	Read-only query evaluation	Writes, locks, migrations
Nobody can read an EXPLAIN plan	Nothing; learn plans first	Everything, until the team can interpret cost

The Harness: A Blind Calibration for SQL Reviewers

The harness below is a condensed version of the one I run for review calibration. It targets Postgres 14 or newer, works on the free server option or any local sandbox, and never executes a write outside a rolled-back transaction. This is not another regression job for generated SQL; it is a calibration harness for the human who approves it.

Create the sandbox schema with python calibrate_reviewer.py init.
Define cases with ground-truth verdicts; the file ships with five, and you should grow the set from real incidents.
Run every case through EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON), wrapping UPDATE and DELETE statements in a transaction that rolls back.
Emit review_pack.json with plans but no verdicts, so the reviewer cannot see the ground truth.
Ask the reviewer to classify each query as correct, slow, or dangerous in a separate file.
Score the verdicts and print accuracy with python calibrate_reviewer.py score review_verdicts.json.

#!/usr/bin/env python3
"""calibrate_reviewer.py — measures how accurately humans review generated SQL.

This is a reviewer calibration harness, not a model benchmark. It runs a
small set of SQL cases through EXPLAIN, hides the ground truth, and scores
the human who reviews them.

Usage:
  python calibrate_reviewer.py init
  python calibrate_reviewer.py run
  python calibrate_reviewer.py score review_verdicts.json
"""

import json
import subprocess
import sys
from pathlib import Path

DSN = "postgresql://localhost:5432/review_sandbox"

SCHEMA = """
CREATE TABLE IF NOT EXISTS customers (
    id INT PRIMARY KEY,
    region TEXT NOT NULL,
    signup_date DATE NOT NULL
);
CREATE TABLE IF NOT EXISTS orders (
    id INT PRIMARY KEY,
    customer_id INT NOT NULL REFERENCES customers(id),
    total NUMERIC(12, 2) NOT NULL,
    status TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL
);
"""

CASES = [
    {
        "id": "ok_indexed_filter",
        "sql": "SELECT id FROM orders WHERE customer_id = 42 AND status = 'paid' "
               "ORDER BY created_at DESC LIMIT 10;",
        "verdict": "correct",
        "note": "Sargable predicate on an indexed foreign key.",
    },
    {
        "id": "missing_where",
        "sql": "UPDATE orders SET status = 'archived';",
        "verdict": "dangerous",
        "note": "No WHERE clause; touches every row in the table.",
    },
    {
        "id": "cartesian_join",
        "sql": "SELECT c.id, o.total FROM customers c JOIN orders o "
               "ON c.region = o.status;",
        "verdict": "dangerous",
        "note": "Join predicate compares unrelated columns.",
    },
    {
        "id": "non_sargable",
        "sql": "SELECT id FROM orders WHERE DATE(created_at) = CURRENT_DATE;",
        "verdict": "slow",
        "note": "Function on the column prevents index usage.",
    },
    {
        "id": "implicit_cast",
        "sql": "SELECT id FROM customers WHERE signup_date = '2026-08-26';",
        "verdict": "correct",
        "note": "Literal comparison is safe and index-friendly.",
    },
]


def psql(sql: str) -> str:
    result = subprocess.run(
        ["psql", DSN, "-At", "-c", sql],
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout.strip()


def explain(sql: str) -> dict:
    if sql.lstrip().upper().startswith(("UPDATE", "DELETE")):
        sql = f"BEGIN; EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) {sql}; ROLLBACK;"
    else:
        sql = f"EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) {sql};"
    for line in psql(sql).splitlines():
        if line.startswith("["):
            return json.loads(line)[0]["Plan"]
    raise RuntimeError("no plan found in EXPLAIN output")


def cmd_init() -> None:
    psql(SCHEMA)
    print("sandbox schema ready")


def cmd_run() -> None:
    pack = []
    for case in CASES:
        plan = explain(case["sql"])
        pack.append(
            {
                "id": case["id"],
                "sql": case["sql"],
                "plan": {
                    "node_type": plan.get("Node Type"),
                    "total_cost": plan.get("Total Cost"),
                    "estimated_rows": plan.get("Plan Rows"),
                },
            }
        )
    Path("review_pack.json").write_text(json.dumps(pack, indent=2))
    print("review_pack.json written; verdicts are hidden until scoring")


def cmd_score(verdicts_path: str) -> None:
    truth = {case["id"]: case["verdict"] for case in CASES}
    human = json.loads(Path(verdicts_path).read_text())
    rows = []
    for case in CASES:
        verdict = human.get(case["id"], "missing")
        rows.append(
            {
                "id": case["id"],
                "human": verdict,
                "truth": truth[case["id"]],
                "match": verdict == truth[case["id"]],
            }
        )
    correct = sum(row["match"] for row in rows)
    print(f"reviewer accuracy: {correct}/{len(rows)}")
    for row in rows:
        marker = "ok " if row["match"] else "MISS"
        print(f"{marker} {row['id']}: human={row['human']}, truth={row['truth']}")


if __name__ == "__main__":
    command = sys.argv[1]
    if command == "init":
        cmd_init()
    elif command == "run":
        cmd_run()
    elif command == "score":
        cmd_score(sys.argv[2])
    else:
        raise SystemExit("usage: calibrate_reviewer.py init|run|score <file>")

Reading the Report

Accuracy alone is not the whole story. A reviewer who marks everything dangerous will score well on a pack full of dangerous cases and uselessly on a pack full of correct ones, so track false positives separately from false negatives. The script prints per-case matches precisely so you can see which bug classes the reviewer misses; a MISS on non_sargable means the team needs more training on index usage, not more tokens. Re-run the same pack monthly to measure decay, then rotate in new cases from real incidents so the pack does not become a memorized exam.

When This Approach Fails

The harness has three honest limits. First, the sandbox does not simulate production: no concurrency, no lock contention, and tiny row counts, so a query that scores well here can still be catastrophic at scale. Second, reviewer accuracy is not reviewer vigilance; a reviewer can score well on known bug classes and miss a novel one. Third, published free tiers can change, so verify the current token allowance and server terms before building a workflow around them.

Do not use this approach if your team has no one who can author ground truth by reading an EXPLAIN plan, because the calibration is only as good as its labels. Do not use it if your production data cannot be mirrored synthetically, since the harness should never run against real customer data. And do not expect the free tier to replace load testing, staging, or a DBA review gate; it measures reviewers, not production.

If you want to see how your own review instincts hold up, fork the script, run it against the free server option, and classify the pack before you read the notes. The verdicts are usually humbling, and that is exactly the point: the reviewer is the last line of defense, and the last line deserves a measurement.

The Slow Query Sentinel: A Zero-Dollar Server That Explains Your Database's Worst SQL

Morgan Li — Tue, 25 Aug 2026 09:06:30 +0000

Last Tuesday, a query that had been fast for months suddenly started eating CPU. The on-call engineer spent forty minutes reading EXPLAIN output before realizing a column type change had killed the index. I have been there too, and it is why I built a sentinel that does not wait to be asked. The tool does not replace a DBA; it shortens the gap between noticing a slow query and understanding why it is slow.

MonkeyCode is an open-source AI coding assistant that currently offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server is not a production environment, but it is an ideal place to run a monitoring tool that only needs to read query statistics. The sentinel I describe below pulls the slowest queries from pg_stat_statements, formats them with their execution plans, and asks a free model to explain the root cause in plain language.

The Architecture of a Self-Explaining Sentinel

A self-explaining sentinel has four parts: a collector, a formatter, a model caller, and a storage layer. The collector reads cumulative statistics, so it does not need to run continuously. The formatter turns raw query text and plan JSON into a prompt that asks for specific advice. The model caller is the only piece that depends on MonkeyCode's free model access, and the storage layer keeps every recommendation for later review.

Step 1: Prepare the Free Server Database

Create a database on the free server and enable pg_stat_statements. The extension ships with standard PostgreSQL, so you only need to adjust shared_preload_libraries and restart the instance. The snippet below also creates a table for storing recommendations, which keeps the output of every model call in one place.

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

CREATE TABLE IF NOT EXISTS query_advice (
    id BIGSERIAL PRIMARY KEY,
    query_id BIGINT,
    query_text TEXT,
    plan JSONB,
    advice TEXT,
    impact_estimate TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Step 2: Write the Collector

The collector queries pg_stat_statements for the top queries by total execution time, then fetches a fresh EXPLAIN (FORMAT JSON) for each one. It writes the raw data to a staging list so the model caller can process them later. Filtering out system queries is important, because you want advice about application SQL, not about internal catalog lookups.

import json
import psycopg2

def collect_slow_queries(dsn: str, limit: int = 10) -> list[dict]:
    with psycopg2.connect(dsn) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT queryid, query, calls, total_exec_time
                FROM pg_stat_statements
                WHERE query NOT LIKE '%%pg_%%'
                ORDER BY total_exec_time DESC
                LIMIT %s
            """, (limit,))
            rows = cur.fetchall()
            items = []
            for queryid, query, calls, total_ms in rows:
                cur.execute("EXPLAIN (FORMAT JSON) " + query)
                plan = cur.fetchone()[0]
                items.append({
                    'query_id': queryid,
                    'query': query,
                    'calls': calls,
                    'total_ms': total_ms,
                    'plan': plan,
                })
    return items

Step 3: Ask the Free Model for an Explanation

This is where the free model access comes in. The function below sends the query text and the plan JSON to a model endpoint and asks for a structured answer. The endpoint and key are placeholders because MonkeyCode's exact interface may change; adapt them to the project's documentation.

import os
import requests

def ask_model(query: str, plan: dict) -> dict:
    prompt = f"""
    Explain why this SQL query is slow and suggest a concrete fix.
    Query: {query}
    Plan: {json.dumps(plan)}
    Respond with JSON: {{"advice": "...", "impact": "high|medium|low"}}
    """
    resp = requests.post(
        os.environ["MODEL_ENDPOINT"],
        headers={"Authorization": f"Bearer {os.environ['MODEL_KEY']}"},
        json={"messages": [{"role": "user", "content": prompt}]},
        timeout=30,
    )
    return resp.json()["choices"][0]["message"]["content"]

A typical response looks like this: the model points out a sequential scan on a large table, suggests a composite index, and marks the impact as high. You do not need to trust it blindly; the value is that it gives you a starting point for your own investigation.

Step 4: Store and Rank the Advice

Parse the model response and insert it into the query_advice table. A simple SELECT then ranks recommendations by the impact field, so the most valuable fixes appear first. This turns a pile of raw plans into a prioritized backlog.

def store_advice(dsn: str, item: dict, advice: dict) -> None:
    with psycopg2.connect(dsn) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                INSERT INTO query_advice (query_id, query_text, plan, advice, impact_estimate)
                VALUES (%s, %s, %s, %s, %s)
            """, (item['query_id'], item['query'], json.dumps(item['plan']),
                  advice['advice'], advice['impact']))

SELECT impact_estimate, advice, created_at
FROM query_advice
ORDER BY CASE impact_estimate WHEN 'high' THEN 0 WHEN 'medium' THEN 1 ELSE 2 END;

Step 5: Schedule and Review

Run the collector and the model caller from a cron job every hour. The free server can handle this load easily, and the free token allowance covers a small number of queries per hour. Review the advice table once a day and apply only the changes that survive your own reasoning.

*/60 * * * * cd /opt/sentinel && python3 run.py

Who Should Not Use This Approach

Do not point this sentinel at a database containing regulated personal data, because the free server is not a production environment. Do not treat model advice as truth; it is a hypothesis that needs a human to verify. Teams that already have a dedicated DBA might find the advice too generic, and teams that cannot tolerate any external call should skip the model step entirely.

The Position, Restated

A zero-dollar server can be more than a test sandbox; it can run a quiet assistant that explains the worst parts of your database. The value is not in the free tokens themselves but in the habit of asking why a query is slow before you rewrite it. If you want to try this pattern, MonkeyCode's free server and free model access are a low-cost way to start, and the open-source project is worth reading before you trust it.

Synthetic Data Is the Missing Layer in Your SQL Eval Harness

Morgan Li — Mon, 24 Aug 2026 19:41:36 +0000

Last spring, a colleague's team shipped an AI-generated query that accidentally exposed row-level customer data from a supposedly anonymized copy of their production database. The anonymization had removed names, but the query's WHERE clause still matched on a combination of attributes that re-identified individuals. That incident taught me a hard rule: any dataset used to evaluate LLM-generated SQL must be synthetic, unless you can prove your real data is both safe and representative.

Most teams I talk to skip this step because they think synthetic data is too much work or too unrealistic. They are wrong on both counts. A free tier with a token allowance and a disposable server can turn synthetic data generation into a routine part of your eval harness, and it costs nothing but a few hours of setup.

In this post, I will argue that synthetic data is not a nice-to-have for SQL model evaluation; it is the foundation. I will show you a concrete pipeline that uses MonkeyCode's free model access and free server option to generate, load, and test against synthetic data, and I will explain where this approach breaks down.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Real Data Is a Broken Test Fixture

Real data carries the exact biases you want your model to unlearn, but it also carries legal and ethical baggage that no test suite should inherit. A production copy, even with columns renamed, still contains statistical fingerprints that can identify individuals, and a generated query that filters on those fingerprints becomes a privacy incident waiting to happen. Beyond compliance, real data is also a poor test oracle because it changes over time, so a regression that passes today may fail tomorrow for reasons unrelated to your model.

The Free Tier as a Synthetic Data Factory

MonkeyCode's free tier includes a 10 million token allowance and a server option that you can treat as a disposable sandbox. That combination is ideal for generating synthetic data, because you can prompt the model to write INSERT statements that match your schema, then execute them on a throwaway SQLite database. The server gives you a clean environment, and the token allowance lets you iterate on the generation prompt without watching a meter.

Here is a minimal Python script that builds a synthetic fixture from a schema definition:

# synthetic_fixture.py — minimal example, not production code
import sqlite3
import random
import string

SCHEMA = '''
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email TEXT NOT NULL,
    country TEXT NOT NULL,
    age INTEGER
);
'''

def fake_email():
    return ''.join(random.choices(string.ascii_lowercase, k=8)) + '@example.com'

def build_fixture(path: str, rows: int = 1000) -> None:
    conn = sqlite3.connect(path)
    conn.executescript(SCHEMA)
    countries = ['US', 'GB', 'DE', 'FR', 'JP']
    for i in range(rows):
        conn.execute(
            'INSERT INTO users (email, country, age) VALUES (?, ?, ?)',
            (fake_email(), random.choice(countries), random.randint(18, 80))
        )
    conn.commit()
    conn.close()

if __name__ == '__main__':
    build_fixture('fixture.sqlite3')

This script is deliberately simple, but it already gives you three properties that real data lacks: controlled cardinality, known distributions, and no privacy risk. You can extend it with edge cases like NULLs, duplicate emails, or extreme ages, and you can regenerate it deterministically with a fixed random seed.

Evaluating Generated SQL on Synthetic Data

Once the fixture exists, you can use the free model access to generate SQL for a natural-language task, then run that SQL against the synthetic database and score the result. The key is to treat the synthetic data as an oracle: you know the exact row counts and distributions, so any deviation from the expected output is a real bug in the generated query.

# eval_generated_sql.py — minimal example, not production code
import sqlite3
import json
import sys

def evaluate(sql_path: str, fixture: str) -> dict:
    conn = sqlite3.connect(fixture)
    conn.execute('PRAGMA query_only = ON')
    cur = conn.cursor()
    sql = open(sql_path).read()
    try:
        cur.execute(sql)
        rows = cur.fetchall()
        return {'ok': True, 'row_count': len(rows)}
    except Exception as exc:
        return {'ok': False, 'error': str(exc)}
    finally:
        conn.close()

if __name__ == '__main__':
    print(json.dumps(evaluate(sys.argv[1], 'fixture.sqlite3')))

The read-only pragma ensures that a generated DELETE or UPDATE fails loudly instead of corrupting your fixture. That loud failure is exactly what you want from a test environment, and it is why the free server is the right place to run this loop.

Limitations and Who Should Skip This

Synthetic data cannot reproduce the performance characteristics of a production database, so it is useless for benchmarking query execution times. It also cannot capture the long-tail of real-world data quality issues, such as inconsistent encodings or subtle cross-column correlations. If your goal is to test index behavior or query plans under realistic load, you need a different approach. But if your goal is to verify that a model-generated query returns the correct logical result, synthetic data is not just acceptable; it is superior to any anonymized copy.

The Bottom Line

A free tier with a token allowance and a disposable server is not a production gift; it is a test infrastructure gift. The most valuable thing you can build with it is a synthetic data pipeline that makes your SQL eval harness reproducible, private, and deterministic. Start with a simple schema, generate a few hundred rows, and see how your favorite model performs when the answer is known in advance.

If you already have an eval harness, add a synthetic fixture to it this weekend. If you do not, build one with the script above and let the free tier do the heavy lifting.

Opinion: Free AI Coding Tokens Are a Model Release, Not a Gift

Morgan Li — Sat, 22 Aug 2026 14:45:43 +0000

Liquid syntax error: Unknown tag 'endraw'

The Night an AI-Generated SQL Query Locked My Production Table

Morgan Li — Fri, 21 Aug 2026 13:15:15 +0000

It was a Saturday evening, and I was confident. A new model had just dropped, and the leaderboard said it was the best thing since window functions. I pointed it at a real migration task, reviewed the SQL briefly, and scheduled it to run at 2 AM when traffic was low. At 3:14 AM, my phone lit up with a pager alert: the orders table was locked, the replica lag had spiked, and every checkout request was timing out. The AI-generated SQL was correct in the sense that it returned the right rows. It was also catastrophically wrong in the sense that it scanned the entire table, held a write lock for eleven minutes, and brought a production service to its knees.

This is not a story about a bad model. It is a story about a bad evaluation strategy, and it is the reason I now treat every AI-generated SQL query as guilty until proven innocent.

The Mistake I Made

The query looked fine in isolation. It joined two tables, filtered on a timestamp column, and returned about four thousand rows. What I did not check was the execution plan, because the test database had a fraction of the production data volume. On a 10,000-row test table, the query planner chose an index scan and the query finished in 40 milliseconds. On a 40-million-row production table, the same query triggered a full table scan, escalated to a table-level lock, and blocked every concurrent write.

The root cause was not the SQL syntax. The root cause was that my evaluation pipeline measured correctness but never measured cost. I compared result sets, confirmed they matched, and shipped it. I never asked how many rows the query touched, whether the planner chose a sensible join order, or what happened under concurrent load.

What I Changed After the Incident

My evaluation workflow now has three distinct gates, and only the first one checks correctness. The second gate checks the execution plan for red flags: sequential scans on large tables, missing index hints, cartesian products, and implicit type conversions. The third gate runs the query against a frozen fixture that mimics production data distribution, not just production schema.

# plan_gate.py — reject SQL with dangerous execution plans
import sqlite3
import sys

DANGEROUS_OPERATIONS = ["SCAN TABLE", "TEMP B-TREE", "CARTESIAN"]

def check_plan(db_path, sql):
    conn = sqlite3.connect(db_path)
    plan = conn.execute(f"EXPLAIN QUERY PLAN {sql}").fetchall()
    for row in plan:
        detail = row[3]
        for marker in DANGEROUS_OPERATIONS:
            if marker in detail:
                print(f"REJECT: {marker} in {detail}")
                return False
    return True

if __name__ == "__main__":
    sql = sys.argv[1]
    ok = check_plan("fixture.db", sql)
    sys.exit(0 if ok else 1)

The plan gate is deliberately conservative. It rejects queries that touch too much data, and it forces a human to read the SQL and decide whether the cost is justified. This single check would have caught my Saturday night disaster, because the production plan would have shown a sequential scan on a table that the test plan indexed.

Building the Defense on a Zero Budget

The full pipeline now runs on free infrastructure. The fixture database is committed to the repository. The evaluation script runs on a schedule, and every model drop triggers a fresh pass through the correctness gate and the plan gate. When both pass, the SQL is sent to a staging environment for a real execution against a data-distribution replica.

The open-source MonkeyCode project provides free model access (10 million tokens) and a free server option, which is how I run this pipeline without paying for API calls or a cloud instance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked their models here, and the pipeline works with any OpenAI-compatible endpoint.

What This Approach Still Cannot Catch

Execution plans catch cost problems, but they do not catch every failure mode. The plan gate misses deadlocks under concurrency, lock contention from long transactions, and performance degradation that only appears at a specific load threshold. It also misses queries that are correct on the fixture but wrong on production data because of NULL distribution or collation differences. For those cases, I rely on a shadow replay: logging production queries and replaying them against a staging copy.

The Lesson

The leaderboard told me the model was smart. The test database told me the SQL was correct. Neither told me the query would lock a production table for eleven minutes. The only thing that would have caught it was a plan gate, a realistic fixture, and a refusal to trust correctness alone. I still use AI-generated SQL every week, but nothing ships without passing all three gates. The cost of that pipeline is zero dollars. The cost of skipping it was a Saturday night I do not want to repeat.

A Free-Server Regression Job for LLM-Generated SQL

Morgan Li — Fri, 14 Aug 2026 11:29:00 +0000

LLM-generated SQL breaks in ways that ordinary unit tests miss. A model can return a syntactically valid query against a schema it has never seen, then a week later return the same-sounding query with an extra join, a wrong date filter, or a column alias that breaks the reporting tool. Teams often respond by adding a paid CI job that provisions a database and runs every prompt through the latest model. That approach works, but it adds cost and setup time before the first useful signal arrives.

This tutorial builds a small nightly regression check that runs against a frozen SQLite fixture database. It uses the free model access and free server option available in MonkeyCode to keep the loop near zero cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness has three parts: a fixture schema, a set of prompt-to-SQL cases with golden outputs, and a runner that calls the model endpoint, executes the generated query inside a rolled-back transaction, and compares a hash of the result columns and rows.

Why a frozen fixture matters

A regression check is only useful when the baseline is stable. A live development database changes constantly, so a failing SQL test could mean the model improved, the data shifted, or an analyst added a column. Those causes are hard to separate. A frozen SQLite fixture removes that variable. Because SQLite is file-based, the fixture can be stored in the repository and rebuilt deterministically before each run.

What the harness does

Creates a schema with a few tables that mirror a typical reporting surface.
Reads a JSONL file containing natural-language prompts and expected SQL.
Sends each prompt to a model endpoint through a thin HTTP client.
Executes the returned SQL inside a transaction and rolls it back so the fixture stays unchanged.
Hashes the result columns and rows and compares them with a golden hash.
Exits with a non-zero code when any case differs.

What the harness does not do

It does not evaluate query performance.
It does not verify business intent across arbitrary new questions.
It does not replace a real database dialect test.

Prerequisites

Python 3.11 or later.
A MonkeyCode account with free model access and a free server option.
The current model endpoint URL and token stored in environment variables.
A small repository with the fixture schema, cases file, and runner script.

The code below is a template. Replace endpoint names, request shapes, and response parsing with the values shown in the current MonkeyCode API documentation.

Step 1: Create the fixture schema

Create schema.sql with a small reporting-like dataset.

CREATE TABLE customer (
  customer_id INTEGER PRIMARY KEY,
  signup_date TEXT NOT NULL,
  country TEXT NOT NULL
);

CREATE TABLE orders (
  order_id INTEGER PRIMARY KEY,
  customer_id INTEGER NOT NULL REFERENCES customer(customer_id),
  amount NUMERIC NOT NULL,
  order_date TEXT NOT NULL
);

INSERT INTO customer VALUES
  (1, '2025-01-10', 'US'),
  (2, '2025-02-14', 'CA'),
  (3, '2025-03-01', 'US');

INSERT INTO orders VALUES
  (101, 1, 120.00, '2025-06-01'),
  (102, 1, 80.50, '2025-06-15'),
  (103, 2, 200.00, '2025-06-20'),
  (104, 3, 15.00, '2025-07-01');

Step 2: Define prompt cases and golden hashes

Create cases.jsonl. Each line contains a prompt, the expected SQL, and a golden hash computed from the expected result columns and rows.

{"id":"us_revenue","prompt":"Find total revenue from US customers ordered by order date ascending","golden_sql":"SELECT SUM(o.amount) AS total_revenue FROM orders o JOIN customer c ON c.customer_id = o.customer_id WHERE c.country = 'US'","golden_hash":"replace-with-computed-hash"}
{"id":"customer_order_count","prompt":"Return each customer and their order count, including customers with no orders","golden_sql":"SELECT c.customer_id, COUNT(o.order_id) AS order_count FROM customer c LEFT JOIN orders o ON o.customer_id = c.customer_id GROUP BY c.customer_id ORDER BY c.customer_id","golden_hash":"replace-with-computed-hash"}

Step 3: Build the model client

Create generate_sql.py.

import os
import httpx

def generate_sql(prompt: str) -> str:
    resp = httpx.post(
        os.environ['MONKEYCODE_API_URL'],
        headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_TOKEN']},
        json={
            'model': os.environ.get('MONKEYCODE_MODEL', 'free-default'),
            'messages': [{'role': 'user', 'content': prompt}],
        },
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    return data['choices'][0]['message']['content'].strip()

The response parsing assumes an OpenAI-compatible choices[0].message.content shape. If the current MonkeyCode endpoint returns a different shape, adjust the client or wrap it with a small adapter rather than changing the rest of the harness.

Step 4: Execute and hash the result

Create runner.py.

import hashlib
import json
import sqlite3
import sys
from pathlib import Path

from generate_sql import generate_sql

def hash_result(columns, rows):
    payload = json.dumps(
        {'columns': columns, 'rows': rows}, sort_keys=True, default=str
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def execute_read_only(db_path: str, sql_text: str):
    conn = sqlite3.connect(db_path)
    cur = conn.cursor()
    cur.execute('BEGIN')
    try:
        cur.execute(sql_text)
        rows = cur.fetchall()
        columns = [description[0] for description in cur.description]
        return columns, rows
    finally:
        conn.rollback()
        conn.close()

def load_cases(cases_path: str):
    cases = []
    for line in Path(cases_path).read_text().splitlines():
        if line.strip():
            cases.append(json.loads(line))
    return cases

def main():
    if len(sys.argv) != 3:
        print('usage: python runner.py <db_path> <cases_path>', file=sys.stderr)
        sys.exit(2)

    db_path, cases_path = sys.argv[1], sys.argv[2]
    failures = []

    for case in load_cases(cases_path):
        generated_sql = generate_sql(case['prompt'])
        columns, rows = execute_read_only(db_path, generated_sql)
        actual_hash = hash_result(columns, rows)

        if actual_hash != case['golden_hash']:
            failures.append({
                'id': case['id'],
                'expected_hash': case['golden_hash'],
                'actual_hash': actual_hash,
                'generated_sql': generated_sql,
            })

    if failures:
        print(json.dumps(failures, indent=2))
        sys.exit(1)

    print('all cases passed')

if __name__ == '__main__':
    main()

The runner does not compare golden_sql to the generated SQL. It compares the executed result hash, which tolerates semantically equivalent query rewrites while still catching output changes.

Step 5: Compute the golden hashes once

Before the nightly run starts, compute the expected hashes against the fixture database using the SQL stored in each case.

# compute_hashes.py
import json
import sys
from runner import execute_read_only, hash_result

def main():
    db_path, cases_path = sys.argv[1], sys.argv[2]
    for case in json.loads(open(cases_path).read()):
        columns, rows = execute_read_only(db_path, case['golden_sql'])
        case['golden_hash'] = hash_result(columns, rows)
        print(json.dumps(case))

if __name__ == '__main__':
    main()

Run it with:

python compute_hashes.py fixture.db cases.jsonl > cases.hashed.jsonl

Then move the hashed file back to cases.jsonl.

Step 6: Schedule the run on the free server

Because the harness is a single Python file and a SQLite fixture, it fits on a small free server. Store the following files in the same directory:

schema.sql
cases.jsonl
generate_sql.py
runner.py
compute_hashes.py

Rebuild the fixture at the start of each run so the server does not depend on persistent local state.

sqlite3 fixture.db < schema.sql
python runner.py fixture.db cases.jsonl

A minimal cron entry can run it every night at 03:00.

0 3 * * * cd /path/to/sql-regression && sqlite3 fixture.db < schema.sql && python runner.py fixture.db cases.jsonl

Replace cd /path/to/sql-regression with the actual free server path. Keep the run quiet by appending > /tmp/sql-regression.log 2>&1 when that is supported.

Decision table: when this fits

Situation	Good fit?	Notes
Fixture has fewer than 10,000 rows	Yes	SQLite handles this quickly on a small server.
Prompts are mostly SELECT statements	Yes	The transaction rollback pattern expects read-only queries.
Queries include vendor-specific dialect features	No	SQLite will not validate Oracle, PostgreSQL, or SQL Server syntax.
You need a signal on every pull request	Possibly	A free server may not be fast enough or always available.
You expect private schema data	No	Keep sensitive schemas inside a controlled environment.
You want a production performance check	No	This harness checks output shape and values, not latency or plans.

Limitations

Free model access and free server resources are limited. Keep the fixture and case list small.
The model endpoint may return multiple SQL statements or explanatory text. Strip the text carefully or wrap the call to extract only the first SQL block.
A hash comparison is strict. Adding a column, changing row order, or changing a number format will fail even if the answer is still useful.
SQLite is not a substitute for the production database engine.
The free server may be ephemeral. Keep all configuration in the repository and rebuild from schema.sql.

Who should not use this approach

Teams that need dialect-specific validation against a real database engine.
Teams with large, complex, or sensitive schemas.
Teams that require sub-second model responses at high request volume.
Teams that already have an expensive, maintained test environment and need more than a lightweight regression signal.

Closing note

A small frozen-fixture harness lowers the cost of noticing when a model's SQL output changes. It is deliberately narrow: one fixture, a few prompts, and a strict result hash. That narrowness makes it easy to reason about and cheap to run. The next step is to add cases for the queries that have caused the most incidents, not to model every possible question.

A New Model Dropped. Is It Actually Good at Your SQL? A 30-Minute Smoke Test

Morgan Li — Mon, 10 Aug 2026 10:38:56 +0000

Liquid syntax error: Unknown tag 'endraw'

Shadow-Gate Your LLM-Generated SQL: A Replay Test Against a Frozen Fixture Database

Morgan Li — Mon, 10 Aug 2026 08:11:39 +0000

In a previous post I built a zero-budget eval harness to score LLM-generated SQL before adopting a prompt. That harness answers one question: "is this generation setup any good?" This post answers a different, more operationally painful one: "the generation setup was good last month — is it still good today, after we tweaked the prompt, swapped the model, or the provider shipped a silent update?"

The failure mode I care about here is semantic drift: the SQL still parses, still runs, still returns rows — but the rows are subtly wrong. A LEFT JOIN quietly becomes an INNER JOIN. A timezone boundary shifts. NULL handling changes. Nothing throws, so your linter and your unit tests on the application code stay green while a dashboard silently lies.

The fix I'll walk through is a shadow gate: a small CI job that replays a fixed suite of analyst questions through your current LLM generation path, executes the resulting SQL against a frozen fixture database, and diffs the result sets against committed golden snapshots. No production data, no warehouse access, no credentials in CI.

The components

A frozen fixture database. A DuckDB file built from CSVs checked into the repo. Frozen means it changes only through reviewed PRs, never as a side effect of the test run.
A question suite. 15–40 real analyst questions with known-correct SQL, curated from actual incidents and code review history.
A generation step. Your real prompt template, filled with the real schema, run against whatever model you currently use.
A diff step. Execute generated SQL on the fixture DB, compare to golden output with explicit rules for column order, float tolerance, and row ordering.

The artifact below is a working skeleton you can adapt.

Step 1: Freeze a fixture database

# build_fixture.py — run once per fixture change, via PR only
import duckdb

con = duckdb.connect("fixture.duckdb")
con.execute("""
    CREATE TABLE orders AS
    SELECT * FROM read_csv_auto('fixtures/orders.csv', header=true);
""")
con.execute("""
    CREATE TABLE customers AS
    SELECT * FROM read_csv_auto('fixtures/customers.csv', header=true);
""")
con.close()

Keep the CSVs small (hundreds of rows) but adversarial: include duplicate customer names, NULL region values, orders exactly on a date boundary, a currency code that appears once. Golden-file tests are only as good as their edge cases — this is where prior production bugs earn their keep.

Step 2: The question suite

# suite.yaml
- id: q001
  question: "Total revenue per region for 2025, excluding cancelled orders"
  golden_sql: |
    SELECT c.region, SUM(o.amount) AS revenue
    FROM orders o JOIN customers c USING (customer_id)
    WHERE o.status <> 'cancelled'
      AND o.order_date >= DATE '2025-01-01'
      AND o.order_date <  DATE '2026-01-01'
    GROUP BY c.region
    ORDER BY c.region;
  order_matters: true

- id: q002
  question: "Customers with no orders, including those with NULL region"
  golden_sql: |
    SELECT c.customer_id, c.region
    FROM customers c
    LEFT JOIN orders o USING (customer_id)
    WHERE o.order_id IS NULL
    ORDER BY c.customer_id;
  order_matters: true

q002 is deliberately a trap: models love converting LEFT JOIN ... IS NULL into NOT IN, which silently drops nothing here but changes semantics the moment customer_id can be NULL in orders. Encode your scars.

Step 3: Generate and replay

For the generation step in CI you need a model endpoint that won't bill you per experiment and doesn't require you to stand up GPU infrastructure. I've been running this class of job through MonkeyCode, which offers free model access and a free server option — that combination maps well onto a CI job that fires a few dozen generation calls per PR and needs a throwaway runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate logic below is provider-agnostic, though — swap the generate() body for whatever endpoint you use, including a local model, and nothing else changes.

# shadow_gate.py
import duckdb, yaml, sys, json

TOLERANCE = 1e-6

def generate(question: str, schema_ddl: str) -> str:
    """Fill your real prompt template; call your model endpoint here.
    Must return raw SQL only (strip markdown fences)."""
    ...  # provider-specific call

def run(con, sql):
    try:
        rel = con.execute(sql)
        cols = [d[0] for d in rel.description]
        return (cols, rel.fetchall(), None)
    except Exception as e:
        return (None, None, str(e))

def rows_equal(golden, candidate, tol=TOLERANCE):
    if len(golden) != len(candidate):
        return False
    for g_row, c_row in zip(golden, candidate):
        for g, c in zip(g_row, c_row):
            if isinstance(g, float) or isinstance(c, float):
                try:
                    if abs(float(g) - float(c)) > tol:
                        return False
                except (TypeError, ValueError):
                    return False
            elif g != c:
                return False
    return True

def main():
    con = duckdb.connect("fixture.duckdb", read_only=True)
    schema_ddl = open("schema.sql").read()
    suite = yaml.safe_load(open("suite.yaml"))
    failures = []

    for case in suite:
        gen_sql = generate(case["question"], schema_ddl)
        g_cols, g_rows, g_err = run(con, case["golden_sql"])
        c_cols, c_rows, c_err = run(con, gen_sql)

        if c_err:
            failures.append({"id": case["id"], "kind": "execution_error",
                             "detail": c_err, "sql": gen_sql})
            continue
        if [c.lower() for c in g_cols] != [c.lower() for c in c_cols]:
            failures.append({"id": case["id"], "kind": "column_mismatch",
                             "detail": f"{g_cols} vs {c_cols}", "sql": gen_sql})
            continue
        g_sorted = g_rows if case.get("order_matters") else sorted(map(str, g_rows))
        c_sorted = c_rows if case.get("order_matters") else sorted(map(str, c_rows))
        if not rows_equal(g_sorted, c_sorted):
            failures.append({"id": case["id"], "kind": "semantic_drift",
                             "detail": "result sets differ", "sql": gen_sql})

    print(json.dumps({"total": len(suite), "failures": failures}, indent=2, default=str))
    sys.exit(1 if failures else 0)

if __name__ == "__main__":
    main()

Three design decisions worth defending:

read_only=True on the connection. Generated SQL is untrusted input. A hallucinated DELETE against your fixture shouldn't be survivable. (For a real warehouse you'd also want a statement timeout and a deny-list on DDL/DML keywords before execution.)
Exact match on columns, tolerance on floats, configurable row order. Anything looser misses real drift; anything stricter drowns you in noise from harmless formatting differences.
The failure report includes the generated SQL. When the gate goes red, the first thing a reviewer needs is the diff between golden and generated query text — not a stack trace.

Step 4: When to run it

Trigger	Why
PR touching the prompt template or schema context	The obvious case; drift source is explicit
Model version change in config	Provider-side behavior changes are the classic silent-drift source
Nightly scheduled run	Catches provider updates that happen without any change on your side
Every application code PR	Overkill unless generation is on the hot path; cost and latency rarely justify it

Gate policy matters more than gate mechanics. My recommendation: semantic_drift failures block merge, execution_error failures block merge only if they didn't occur on the previous nightly baseline (a model that occasionally emits unparseable SQL may be tolerable in a human-in-the-loop workflow, but a regression in parse rate is not).

Limitations and who should skip this

It tests equivalence to golden SQL, not correctness of intent. If your golden SQL encodes a wrong business rule, the gate faithfully protects a wrong answer. Golden queries need the same review rigor as production code.
Sampling cost. A single generation per question will flake on borderline cases. For a tighter signal, generate k=3 samples per question and require a majority to match — which triples your model calls per run, so budget accordingly.
Frozen fixtures age. When the real schema evolves, fixture and goldens must evolve in the same PR, or the gate tests a database that no longer exists.
This is not a substitute for an adoption-time eval. The eval harness scores candidates before you pick a generation setup; the shadow gate watches the setup you picked. They answer different questions and you probably want both — but if you can only build one and your generation setup is stable, build the gate.
Skip this entirely if generated SQL never runs unattended, if you have fewer than ~10 recurring question patterns (manual review is cheaper), or if your queries are trivially re-derived from an ORM (there's no free-form generation surface to drift).

Closing

The uncomfortable truth about LLM-generated SQL is that "it still runs" is a nearly meaningless health signal. A replay gate against a frozen fixture is the cheapest way I know to convert silent semantic drift into a loud, reviewable diff. If you want to try the pattern without provisioning anything, MonkeyCode's free model access and free server tier is a low-friction place to host the generation step while you find out whether your suite catches anything — mine caught a JOIN-flavor regression within the first week, which paid for the setup effort on the spot.