I Stopped Scoring Completions. I Started Scoring Invariants.

The cheapest completion I got this week compiled on the first try and still stole money from a fake ledger. That is not a vibe check. That is a failure class, and most "did the model write code?" scoreboards never look for it.

If your eval rewards a fenced Python block, you are grading costumes. Compile-clean and correct are not the same sport. Why do we keep pretending they are?

I wanted a probe I could rerun on a free model endpoint without arguing with the output. Not another chat log. A small contract: extract the function, run it against invariants, and classify the miss. Looks-like-code is theater. An invariant is a receipt.

The analogy I kept coming back to is a bridge inspection that only checks whether the blueprint is a PDF. Sure, it is a document. Would you drive on it?

The question the probe actually asks

I am not asking whether a free model can "code." That question is soggy. I am asking: for a function with a crisp property, does the completion preserve the property, or does it emit something that imports, types, and still lies?

Cheap code is still debt if the invariant fails. The trend conversation this week is about AI making code inexpensive. Fine. Inexpensive according to whom? Tokens are not the bill. The bill is the silent wrongness you merge because the file parsed.

So I wrote a harness that treats the model like an untrusted intern. The intern may use a free server. I still want a number I can defend.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I pointed the HTTP client at MonkeyCode's free model access and free server option because I needed a backend that would not turn a probe loop into an invoice. The probe does not care about the brand on the box. It cares whether unique_stable keeps order after it drops duplicates.

A tiny corpus with teeth

Three functions. That is the whole exam. I kept it mean and local.

clamp has an obvious implementation and an obvious cheat: clamp only on the low side and hope nobody sends a high outlier. unique_stable is where models fall in love with set(). A set is unique. A set is also a blender. running_balance is the ledger. Credits and debits come in. The balance must never go negative. Plenty of completions sum the list and smile. Summing is not an invariant. Overdraft is.

I do not need Hypothesis for this. A for loop and a few adversarial cases beat a vibes prompt. If your properties need a research lab, you will not rerun them at 11 p.m. I rerun things that fit in one file.

Here is the corpus the probe scores. Copy it. Break it. Add a fourth function tomorrow if you want, but do not start by boiling the ocean.

# invariant_probe.py
from __future__ import annotations

import ast
import json
import os
import re
import textwrap
import urllib.error
import urllib.request
from dataclasses import dataclass, asdict
from typing import Callable

PROMPTS = {
    "clamp": textwrap.dedent(
        '''
        Write a Python function clamp(n, lo, hi) that returns n limited to [lo, hi].
        Assume lo <= hi. Return only a fenced python block. No tests, no prose.
        '''
    ).strip(),
    "unique_stable": textwrap.dedent(
        '''
        Write unique_stable(items) that drops duplicates and keeps first-seen order.
        items is a list of hashable values. Return only a fenced python block.
        '''
    ).strip(),
    "running_balance": textwrap.dedent(
        '''
        Write running_balance(txns) where txns is a list of ints (credits positive,
        debits negative). Return the final balance. If any prefix sum would go below
        zero, raise ValueError. Return only a fenced python block.
        '''
    ).strip(),
}

The oracles live next to the prompts on purpose. If you let the model write the tests, you are grading a student who brought their own answer key. I have watched that movie. The ending is always "green CI, wrong product."

def properties_clamp(fn: Callable) -> None:
    assert fn(5, 0, 10) == 5
    assert fn(-2, 0, 10) == 0
    assert fn(99, 0, 10) == 10
    assert fn(0, 0, 0) == 0

def properties_unique_stable(fn: Callable) -> None:
    assert fn([3, 1, 3, 2, 1]) == [3, 1, 2]
    assert fn([]) == []
    assert fn(["a", "a", "b"]) == ["a", "b"]
    # The trap: set() shuffles or sorts on some versions / some luck.
    sample = ["z", "b", "z", "a", "b"]
    assert fn(sample) == ["z", "b", "a"]

def properties_running_balance(fn: Callable) -> None:
    assert fn([10, -3, -2]) == 5
    assert fn([]) == 0
    try:
        fn([5, -9])
    except ValueError:
        pass
    else:
        raise AssertionError("overdraft must raise ValueError")
    # Prefix 4, then -1, then +10 is legal. Naive "sum first" still passes this.
    assert fn([4, -1, 10]) == 13

ORACLES = {
    "clamp": properties_clamp,
    "unique_stable": properties_unique_stable,
    "running_balance": properties_running_balance,
}

Notice running_balance has a case that a total-sum implementation survives. That is deliberate. A probe that only punches the obvious hole trains you to feel safe. I wanted one hole that looks like success.

Extract, load, accuse

Free endpoints ramble. They apologize. They wrap the function in a novel. The probe has to be ruder than the model.

FENCE = re.compile(r"```

(?:python)?\s*([\s\S]*?)

```", re.I)

def extract_python(text: str) -> str:
    m = FENCE.search(text or "")
    blob = m.group(1) if m else (text or "")
    blob = blob.strip()
    if not blob:
        raise ValueError("empty completion")
    ast.parse(blob)
    return blob

def load_fn(source: str, name: str) -> Callable:
    ns: dict = {}
    exec(source, ns, ns)  # local probe only; never exec untrusted net code in prod
    if name not in ns or not callable(ns[name]):
        raise ValueError(f"{name} missing")
    return ns[name]

Yes, exec is a loaded word. This file is a bench, not a SaaS. I run it on my machine against completions I just fetched. If you paste this into a multi-tenant worker, you are the incident. The limitation section is not decoration.

The HTTP bit is boring on purpose. OpenAI-shaped JSON, one timeout, no retry heroics. I already burned a week on retry folklore in another post. This probe assumes the request either returns text or it does not.

@dataclass
class Verdict:
    task: str
    kind: str  # parse | missing_fn | property | pass | http
    detail: str

def complete(prompt: str) -> str:
    url = os.environ["LLM_BASE_URL"].rstrip("/") + "/chat/completions"
    body = json.dumps({
        "model": os.environ.get("LLM_MODEL", "default"),
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        url, data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get("LLM_API_KEY", "local"),
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    return payload["choices"][0]["message"]["content"]

I leave LLM_MODEL as an env var. I am not going to pretend I know which name your free server wants this week. Names rot. Properties do not.

The score is a kind, not a trophy

def judge(task: str, source: str) -> Verdict:
    try:
        fn = load_fn(source, task)
    except SyntaxError as e:
        return Verdict(task, "parse", str(e))
    except ValueError as e:
        return Verdict(task, "missing_fn", str(e))
    try:
        ORACLES[task](fn)
    except AssertionError as e:
        return Verdict(task, "property", str(e) or "assert failed")
    except Exception as e:
        return Verdict(task, "property", f"{type(e).__name__}: {e}")
    return Verdict(task, "pass", "ok")

def main() -> None:
    rows = []
    for task, prompt in PROMPTS.items():
        try:
            text = complete(prompt)
            source = extract_python(text)
            rows.append(asdict(judge(task, source)))
        except urllib.error.URLError as e:
            rows.append(asdict(Verdict(task, "http", str(e))))
        except Exception as e:
            rows.append(asdict(Verdict(task, "parse", str(e))))
    print(json.dumps(rows, indent=2))

if __name__ == "__main__":
    main()

Run it like a grown-up script, not like a demo GIF.

export LLM_BASE_URL="https://YOUR_FREE_ENDPOINT/v1"
export LLM_API_KEY="local"
python invariant_probe.py

What do you do with the JSON? You do not average it into a leaderboard and go home. You read the kind field. parse means the costume tore. missing_fn means it wrote a helper and forgot the name you asked for. property is the interesting one. That is the intern who compiled, nodded, and overdrafted the ledger.

What I actually learned by running it

I planted three canned completions first, because a probe that cannot catch a bug you wrote by hand is a mood ring. A clamp that only does max(n, lo). A unique_stable that returns list(set(items)). A running_balance that return sum(txns) and never looks at a prefix. The probe yelled at all three. The correct fixtures stayed quiet. Good. The instrument blinks.

Then I aimed the same prompts at a free model server. I am not going to dress one evening of traffic up as a benchmark. I will tell you the shape of the misses, because the shape repeated.

Short numeric clamp was the easy room. Bounded scalars are catnip. The model often passed. That is the "performs well" cell in my head: tiny, locally checkable, no hidden state, no order, no "unless." If your workload is that room, a free completion is a reasonable first draft. You still run the oracle. You just expect more pass rows.

unique_stable is where the blender shows up. Why do models love sets? Because uniqueness is the slogan and order is the fine print. The function looks adult. The property fails on ['z', 'b', 'z', 'a', 'b']. If your eval used assert len(fn(x)) == len(set(x)), you would have stamped it correct. That eval is how debt gets a green check.

running_balance split into two lies. Lie one: no ValueError at all, just a negative number, because the prompt's raise clause is easy to "forget" when the model is busy being helpful. Lie two: it raises on a negative transaction instead of a negative prefix. A debit of -3 on a balance of 10 is legal. A probe that only sends [-1] cannot see the difference. I almost shipped that weaker oracle. Glad I did not.

HTTP misses happened too. Timeouts, empty content, a 200 with a smile and no fence. I bucket those as http or parse and I refuse to call them "the model cannot code." The server coughed. Different organ.

So where does a free model plus a free server hold? Narrow functions, sharp types, properties you can write in ten lines. Where does it break? Stability, prefixes, exceptions as part of the contract, anything you would mark "please don't get cute" on a junior's PR. The completion is not evil. It is optimistic. Optimism is not an invariant.

Who should not use this

Do not use this as a merge gate for production agents. Three functions are a flashlight, not a type system. Do not exec model output on a shared host. Do not treat temperature 0 as a vow of determinism; I set it so reruns are less noisy, not because the server swore an oath. Do not score only the tasks you already know the model can do. That is how you publish a 90 and still ship the blender.

If you need certified numerics, crypto, or anything that moves real money, this probe is a toy. Use it to shame your eval, not to bless a model.

And if you only wanted a title that says the stack is magic, skip the file. The file will insult you. That is the feature.

The bill you actually pay

Technical debt did not wait for cheap code. Cheap code just prints it faster. A free server makes the printing free. The invariant is still on you.

I keep the probe next to the prompts because I am tired of arguments that start with "it looked right." Looking right is how the ledger goes negative. Run the property. Read the kind. Then decide whether the completion was a draft or a costume.