Five Model-Eval Myths Developers Repeat. Here's the Probe That Settles Them.

AI promoted every developer to reviewer. Nobody tested the reviewer. That's the hot take this week. I mostly agree.

But I'd add a second problem. We measure models with broken tools. We repeat myths about measurement. Then we blame the model.

This isn't about server uptime or speed. It's about how we judge model output. I run my eval experiments on MonkeyCode's free model server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The endpoint matters less than the method.

These five myths cost me real debugging hours. Each one has a claim, evidence, and a corrected mental model. One probe settles all of them. Sound familiar?

Why this matters now

AI review tools are everywhere. Code reviewers, PR bots, test generators. The community keeps asking who tests the reviewer.

Fair question. My answer starts with the measuring stick. If your eval is broken, your review is broken. Fix the eval first.

Myth 1: A benchmark score predicts my workload

The claim. "It scores 90% on a public benchmark. It will crush my codebase."

The evidence. Benchmarks test generic tasks. Your repo has private conventions. Your JSON schemas are weird. Your prompts are specific. A model can ace a public leaderboard and still mangle your output format. I've watched it happen.

The corrected mental model. Benchmarks rank models on their turf. Canaries rank them on yours. Build a canary set of five to ten real tasks from your repo. Grade them yourself. That number predicts your workload. The leaderboard number doesn't.

Myth 2: Temperature 0 means deterministic output

The claim. "Set temperature to 0. You get the same answer every time."

The evidence. Sampling is not the only source of randomness. Batching changes results. Kernel selection changes results. Server load changes results. Run the same prompt ten times at temperature 0. Count unique outputs. You will often see more than one.

The corrected mental model. Temperature 0 reduces variance. It does not eliminate it. Treat exact retries as a bonus, not a guarantee.

Myth 3: One good response means the model is reliable

The claim. "It solved my task once. Ship it."

The evidence. One success is a coin flip. You need a pass rate. Run the task ten times. Grade each output. Count passes. Nine out of ten is different from five out of ten. Your confidence should be too.

The corrected mental model. Judge by pass rate over repeated runs. Not by your best run. Free model access makes this cheap. Ten small calls cost almost nothing. Spend the calls. Save the debugging.

Myth 4: More context always helps

The claim. "Feed it the whole repo. More information. Better answer."

The evidence. Context is a budget. Irrelevant files crowd the attention window. Noise drowns the signal. Test the same task with minimal context. Then test it with a bloated dump. Compare the grades.

The corrected mental model. I've seen minimal context win more often than not. Trim context like you trim logs. Only ship what the task needs.

Myth 5: The model is the only variable in your eval

The claim. "Results changed. The model drifted."

The evidence. Your harness drifts too. Prompt wording changes. Parsing logic changes. Library versions change. Even test order changes. An eval measures the whole system. The model is one component.

The corrected mental model. Blaming the model first is lazy. Freeze your harness. Change one thing at a time. Measure the delta. Then point fingers.

The probe: one script, five myths

Here's the probe I run. It settles all five myths with your data, not mine.

# eval_myth_probe.py
"""Settle five eval myths with your own data.

Setup:
    export LLM_BASE_URL="https://your-server.example/v1"
    export LLM_API_KEY="your-key"
    export LLM_MODEL="your-model"

Run:
    python eval_myth_probe.py task.txt
"""

import os
import sys

from openai import OpenAI

client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],
    api_key=os.environ["LLM_API_KEY"],
)
MODEL = os.environ["LLM_MODEL"]


def call(prompt: str, temperature: float = 0.0) -> str:
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return response.choices[0].message.content


def grade(output: str) -> bool:
    """Edit this function. Return True when the output is acceptable."""
    return "TODO" not in output


def main() -> None:
    task = open(sys.argv[1], encoding="utf-8").read().strip()

    # Myth 2: temperature 0 is not always deterministic.
    outputs = [call(task) for _ in range(10)]
    unique = len(set(outputs))
    print(f"[myth-2] unique outputs at temp 0: {unique}/10")

    # Myth 3: one good run is not a pass rate.
    passed = sum(1 for output in outputs if grade(output))
    print(f"[myth-3] pass rate: {passed}/10")

    # Myth 4: more context is not always better.
    bloated = "Ignored context:\n" + ("x" * 4000) + "\n\nTask:\n" + task
    minimal_output = call(task)
    bloated_output = call(bloated)
    print(f"[myth-4] minimal context graded: {grade(minimal_output)}")
    print(f"[myth-4] bloated context graded: {grade(bloated_output)}")


if __name__ == "__main__":
    main()

What it does:

• Runs the same prompt ten times at temperature 0. Counts unique outputs. That's myth 2.
• Grades each output with your grade() function. That's myth 3.
• Compares minimal context against a bloated dump. That's myth 4.
• Keeps every other variable frozen. That's myth 5.
• Your task.txt file is the canary. That's myth 1.

Edit grade(). It's the only part that knows your task. Everything else is generic.

Why ten runs? Because one run proves nothing. A pass rate needs a sample.

Most free model servers expose an OpenAI-compatible endpoint. The probe assumes yours does. If not, swap the client for your SDK. I run this probe against MonkeyCode's free server. The numbers you get will be yours, not mine.

How to read the report

Probe output	What it means	Next step
1 unique output in 10	Deterministic on this task	Exact retries are safe here
5+ unique outputs	High variance	Add a grader. Stop eyeballing
Pass rate 9-10/10	Reliable on this task	Promote to a bigger canary set
Pass rate 5/10 or lower	Unreliable	Change prompt, context, or model
Bloated context wins	Context helps here	Keep context, cut the noise
Minimal context wins	Context pollution	Feed only relevant files

Limitations

This probe is not a benchmark. It measures one task on one endpoint on one day.

It doesn't test latency. It doesn't test streaming. It doesn't test cost. It doesn't test safety.

Don't use it to pick a production model. Don't use it for security decisions. Don't use it to compare models you haven't named.

Who should not use this approach:

• Teams with labeled datasets and formal eval needs.
• Anyone who can't write a grade() function.
• Anyone expecting a permanent verdict. Free endpoints change. Re-run the probe.

The corrected mental model

Here's what I believe now.

Benchmarks rank models on their turf. Canaries rank them on yours.

Determinism is a property of the whole pipeline. Not one parameter.

Pass rates beat single successes. Always.

Context is a budget. Spend it like one.

Your eval measures your whole system. Isolate variables before blaming the model.

Run the probe. Keep the myths out of your review loop. Your reviews will thank you.