Free Endpoint First or Self-Hosted First? Deciding Where an Agent Drafts SQL

Most teams that wire an LLM into a SQL drafting loop spend their first week on prompts, and their second week discovering that the binding constraint was never the prompt. The real constraint is usually placement: which backend writes the candidate query, what it is allowed to see, and what happens when that backend rate-limits you mid-batch. This article works through that decision as a structured debate between two defensible positions, then reduces it to a decision rule you can test on your own corpus.

Illustrative scenario, labeled as such: a team runs an internal agent that turns analyst questions into read-only SQL against a warehouse of roughly a few hundred tables. They have a graded corpus of past questions with human-accepted answers, and a rule-based second reviewer that checks the candidate query against a fixed rubric before anyone runs it. No production numbers are claimed anywhere below; every metric is something you would collect yourself.

Position A: start on free model access and a free server

The first position holds that the cheapest correct first step is to run the whole harness on infrastructure somebody else pays for. Free model access removes the fixed cost of a GPU box, so the team can spend its early budget on the evaluation corpus instead of on provisioning. A free server option also shortens the loop from "we have a hypothesis about draft quality" to "we have a table of runs," which matters more than raw latency at this stage.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The brief accompanying this article described free model access and a free server option; I have not independently measured either capability. Treat both as claims to verify on the official signup page before you size a batch, because quotas, model lists, and server specifications change without notice.

The honest weakness of Position A is the data path: a hosted endpoint receives whatever you put in the prompt, and schema text can be fine while row samples are not. The second weakness is nondeterminism of availability, since a free tier can throttle or shed load exactly when your overnight batch is running. Neither weakness is fatal, but both must be designed for rather than discovered.

Position B: start on a self-contained backend

The second position argues that placement should follow data classification, not cost. If the drafting prompt needs row samples to disambiguate columns, then shipping those samples to a third-party endpoint is a review item, not a convenience trade. Self-hosting also gives you a pinned model identity, which makes a regression in acceptance rate attributable to your prompt change instead of to someone else's silent rollout.

The weakness here is equally concrete: a self-hosted backend is a second system to operate, and its quality at small sizes may be worse than a larger hosted model, which biases your corpus evaluation toward pessimism. Teams frequently under-invest in the corpus while over-investing in the box.

Where the debate actually resolves

Both positions agree on the artifact; they disagree on the ordering. The tie-breaker is not vibes about hosting, it is three measurable properties of your own loop: the egress surface (which fields leave your boundary), the variance of acceptance across repeated runs, and the unit cost per accepted query including escalations. Cost per draft is the wrong denominator, because a cheap draft that fails twice and escalates is more expensive than an honest first attempt.

Step 1: Fix the corpus and the acceptance rule before comparing backends

Freeze a set of questions with known-good SQL, and define acceptance as a boolean produced by the same reviewer for every backend. If your reviewer is a rubric-based checker, keep the rubric byte-identical across runs; if it is a second model, record its identity alongside each verdict.

# illustrative, not executed here
from dataclasses import dataclass

@dataclass(frozen=True)
class Case:
    query_id: str
    question: str
    gold_sql: str
    tables: tuple[str, ...]
    needs_row_samples: bool   # drives the egress decision below

Step 2: Instrument the harness so placement is a variable

Wrap every backend behind one interface so the only thing that changes between arms is the endpoint. Log tokens, latency, escalation, and the exact set of field names you transmitted.

@dataclass
class DraftRun:
    query_id: str
    backend: str          # "hosted-free" | "self-hosted"
    accepted: bool
    escalated: bool       # a second, stronger attempt was needed
    latency_ms: float
    prompt_tokens: int
    completion_tokens: int
    egress_fields: frozenset[str]

def draft(case: Case, backend: str) -> DraftRun:
    """Call the endpoint, grade the result with YOUR reviewer, return the record."""
    raise NotImplementedError  # wire your client and rubric here

Step 3: Put a hard ceiling on the batch, and fail closed

Unbounded retries are how a free tier experiment turns into a surprise bill or a rate-limit cascade. Charge every attempt against a budget object and halt the batch when it is exhausted; do not catch the exception and continue.

class Budget:
    def __init__(self, ceiling_tokens: int) -> None:
        self.ceiling = ceiling_tokens
        self.spent = 0

    def charge(self, tokens: int) -> None:
        if self.spent + tokens > self.ceiling:
            raise RuntimeError("draft budget exhausted; stop the batch")
        self.spent += tokens

If the endpoint answers 429, stop and record it as an observation about the arm, not as noise to retry around. Rate-limit behaviour is part of what you are comparing, and a batch that only completes because of aggressive retries tells you nothing about production.

Step 4: Compare on cost per accepted query

Compute the denominator honestly by counting accepted queries, not attempts, and include any paid fallback in the numerator.

def cost_per_accepted(runs, paid_prices=None):
    """paid_prices: {"backend": (usd_per_1k_prompt, usd_per_1k_completion)}; free arms are 0.0."""
    accepted = {r.query_id for r in runs if r.accepted}
    if not accepted:
        return float("inf")
    total = 0.0
    for r in runs:
        p = (paid_prices or {}).get(r.backend)
        if p:
            total += r.prompt_tokens / 1000 * p[0] + r.completion_tokens / 1000 * p[1]
    return total / len(accepted)

The decision table

Run the corpus once per arm and read the table left to right; the first failing row decides, which prevents cost from silently outvoting a data constraint.

Check	Question to answer from your logs	If it fails
Egress	Do any egress_fields from needs_row_samples cases cross your boundary?	Self-hosted first, regardless of cost
Reproducibility	Does acceptance swing beyond your tolerance across repeated runs of the same arm?	Pin the model or raise sample count before comparing
Rate-limit survival	Does the arm finish a full corpus pass without a 429 halt?	Self-hosted first, or split the batch
Cost per accepted query	Is the free arm cheaper after counting escalations?	Keep the free arm, but cap escalations
Operability	Can one engineer redeploy the arm after a failure?	Free arm if the alternative is an unowned box

Limitations and who should not use this

This decision rule assumes you already have a graded corpus; without one, both positions are guesswork and the comparison will just measure prompt luck. Teams under regulatory constraints that forbid sending row samples anywhere should treat the debate as settled and start self-hosted. Teams that need a contractual p95 latency for interactive drafting should not run the first arm on a free tier whose throttling behaviour is outside your control.

Finally, if your volume is a handful of queries per day, the evaluation harness costs more than either option saves; write the queries by hand and revisit when the corpus is large enough to be worth grading. The useful output of this exercise is not a winner, it is a log format and a decision table you can re-run when a quota, a model, or a data policy changes.

If you want to try the free-endpoint arm without provisioning anything, the free model access and free server option are the parts worth checking first — and verify the current limits on the official site before you plan a batch around them.