I Built a 40-Minute Evaluation for Free Model Endpoints. Here's the Scorecard.

Free model endpoints are seductive. Zero cost. Zero setup. Zero reason to trust them. I don't trust demos. I trust failure modes. So I built a small evaluation harness. It tests one thing: can a free model endpoint gate a pull request for secrets? This is not a benchmark. It's a repeatable experiment. You can run it in an afternoon.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model endpoint and the free server option for the test. No quotas. No hardware claims. Just a harness and a rubric.

Why I stopped trusting free endpoints

Free endpoints look great in a demo. You paste a diff. The model finds the secret. Everyone claps. Then you wire it into CI. The JSON breaks. The latency spikes. The model misses a private key. The demo didn't show that. An evaluation will.

The experiment

I designed a 40-minute test. It answers one question: where does the free endpoint perform well, and where does it break? The dataset is 30 synthetic diffs. Fifteen contain real-looking secrets. Fifteen are clean. Each diff is small. Each diff has one clear change.

The prompt is strict. The model must return JSON. No prose. No apologies. Just a verdict.

# eval_secret_gate.py
# Simplified harness. Adapt to your client SDK.
import json, time

def classify(client, diff: str) -> dict:
    prompt = f"""
You are a secret scanner for code review.
Return ONLY JSON with this shape:
{{"contains_secret": true, "line": 12, "type": "aws_access_key"}}
Diff:
{diff}
"""
    start = time.time()
    response = client.complete(
        prompt,
        model="free",
        server="free",   # free server option
    )
    latency = time.time() - start
    return {"latency": latency, "raw": response}

def evaluate(client, diffs, runs=3):
    for i, diff in enumerate(diffs):
        for run in range(runs):
            yield i, run, classify(client, diff)

The harness is deliberately small. It measures five things. Accuracy. JSON validity. Latency. Variance. Failure modes.

The rubric

Every answer gets one of five labels.

Label	Meaning
True positive	Secret found, line and type correct
False positive	Clean diff flagged as secret
False negative	Secret missed
Parse error	Model returned prose or broken JSON
Timeout	No answer before the cutoff

A good gate needs high recall. A false negative ships a secret. A false positive blocks a merge. Both are expensive. The harness makes them visible.

What the output looks like

After the runs, the harness prints a scorecard. Here's the shape.

Diff	Run 1	Run 2	Run 3	Verdict
diff_01	TP	TP	TP	stable
diff_02	FP	FP	TN	unstable
diff_03	parse	parse	TP	broken
diff_04	FN	FN	FN	consistent miss

This is a template, not a benchmark. Fill it with your own numbers. The pattern tells you more than the average. A consistent miss is worse than a parse error. A parse error is visible. A miss is silent.

Where to look for strengths

The harness will show you where the endpoint performs well. Start with simple patterns. AWS access keys. GitHub tokens. PEM blocks. These have strong signals. If the endpoint handles these, you have a useful gate. Short diffs help. One file. One change. No context to confuse it.

The free server option also matters. You don't need a GPU. You don't need a queue. You send requests and wait. For a small team, that's enough.

Where to expect breakage

Watch for three failure modes. Long diffs are the first. The model can lose context. It starts guessing. It flags a base64 string as a secret. Or it misses the secret in line 400.

JSON drift is the second. The model wants to explain itself. It returns prose. It adds a period. It wraps the JSON in a code block. Your parser breaks. Your CI fails. The demo never showed that.

Variance is the third. The same diff can get different answers. Run one says true. Run two says false. Run three says true. A single run is a coin flip. You need agreement.

The decision table

Use this table when you run the harness.

Observation	Action
Parse rate below 95%	Add schema validation and one retry
False negatives on PEM blocks	Add a regex pre-filter before the model
p95 latency over 10 seconds	Move to an async queue, not blocking CI
Same diff flips across runs	Require two-of-three agreement
False positives on base64	Add an allowlist or a wider context window

This table is the real artifact. It turns a vague "AI is unreliable" into a decision. You can automate each row.

Turning the scorecard into a CI gate

Once you have the scorecard, you can build a gate. The gate runs only on changed files. It calls the endpoint. It parses the JSON. It blocks the merge on a true positive. It logs false positives for review. It never blocks on a parse error. It retries once.

def gate(diff: str) -> str:
    result = classify(client, diff)
    if not valid_json(result["raw"]):
        return "retry"
    if result["contains_secret"]:
        return "block"
    return "pass"

This is the minimal version. Add two-of-three agreement for unstable diffs. Add a regex pre-filter for known patterns. Add a human review queue for uncertain cases.

Limitations and who should not use this

This harness is not a benchmark. It uses synthetic diffs. Real diffs are messier. Real secrets are nested in larger changes. Real teams have different tolerances.

Do not use this approach for compliance. Do not use it as your only secret scanner. Do not use it on a high-throughput CI without a queue. The free endpoint is a helper, not a guarantee.

Who should use it? Small teams. Side projects. Teams that want a second pair of eyes on a PR. Teams that can tolerate a false positive now and then.

The takeaway

Free model endpoints are not magic. They are tools with failure modes. The failure modes are predictable. That means they are testable. That means you can build a gate around them.

Run the harness. Print the table. Find the breakage. Then decide if the free tier is worth it. The scorecard will tell you.