DEV Community

Jordan Huang
Jordan Huang

Posted on

FAQ: Five Eval Claims Your Free-Model Trace Cannot Support

#discuss #ai #python #testing

Is that screenshot an evaluation report? Or is it only a long chat?

I keep getting traces that look official. Green checks. Token counters. A model announcing that tests passed. Then somebody pastes the window into a design doc.

That window is not an eval report. It is a conversation wearing metadata. This FAQ names five claims I still hear. Each claim gets evidence, then a corrected mental model.

You also get a tiny harness. Treat it as a proposal. I am not selling a leaderboard.

Why this gap keeps spreading

Measurement talk is loud again this week. People argue about tests that models already game. Local traces still get treated like lab certificates.

Did the agent finish a task in a free session? Fine. Did you freeze fixtures and assertions first? That is the real question.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. I treat both as a scratch pad for drafting a harness. I do not treat them as proof. No model names. No quota theater. No invented p50.

Myth 1: The completed trace already measured quality

Claim: The run finished, so quality was measured.

Evidence you actually hold: A log of tool calls and prose. Maybe a diff. Maybe nothing scored.

Corrected model: Evaluation needs frozen inputs, machine-checked assertions, and an environment receipt.

Ask these before you quote a pass rate:

  • Which fixture files were hashed before the agent started?
  • Which commands fail without a human nodding along?
  • Which machine facts can you rehydrate next week?

If those answers are empty, stop quoting scores. You are narrating a session.

# proposal: gate any "eval" label on three files
test -f fixtures/input.md || echo "missing fixture"
test -f tests/test_eval.py || echo "missing assertions"
test -f env/receipt.json || echo "missing receipt"
Enter fullscreen mode Exit fullscreen mode

A finished trace is raw material, not a result. Save it. Do not crown it.

Myth 2: Token volume means the eval was serious

Claim: We spent a pile of free tokens, so the study was real.

Evidence you actually hold: Consumption. That is an accounting figure, not accuracy.

Corrected model: Tokens measure burn. They do not measure fitness, safety, or latency.

A cheap failing test can be excellent evidence. A huge chat can prove a vibe and nothing else. Would you accept invoice size as a unit test?

Keep the columns honest:

assertions_passed / assertions_total  -> quality signal
tokens_in + tokens_out                -> cost signal
wall_clock_seconds                    -> time signal
Enter fullscreen mode Exit fullscreen mode

Never fold those into one vanity index. If deleting the assertions still leaves your "score," you were scoring spend.

# proposal: refuse mixed dashboards
if grep -q tokens artifacts/summary.csv && ! test -f artifacts/pytest.out; then
  echo "spend without assertions is not an eval"
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode

Myth 3: The complimentary server is a pinned lab

Claim: It ran on the free box, so the environment is known.

Evidence you actually hold: A borrowed machine. Images move. Processes die. Limits appear without a press release.

Corrected model: A lab is something you can rebuild from notes. Can you rebuild this one from memory? Probably not.

Write a receipt before the agent touches files. Fill it from commands, never from vibes.

# proposal: capture only facts the shell can prove
mkdir -p env artifacts
{
  echo "git_head=$(git rev-parse HEAD 2>/dev/null || echo none)"
  echo "python=$(python3 --version 2>/dev/null || echo none)"
  echo "cwd=$(pwd)"
  echo "fixture_hash=$(sha256sum fixtures/* 2>/dev/null | sha256sum)"
  echo "ran_at_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
} > env/receipt.txt
cat env/receipt.txt
Enter fullscreen mode Exit fullscreen mode

If the free server later goes away, the receipt still speaks. The browser tab will not. This is about scoring, not about promoting a branch.

Myth 4: A confident transcript is a test suite

Claim: The model said the code looks good. Close enough for a green board.

Evidence you actually hold: A review. Reviews help. They are not suites.

Corrected model: A suite fails when you are not in the room. If you must watch it, it is a demo.

Here is an unexecuted pytest sketch. Copy it only after you add real fixtures.

# eval_smoke.py — proposal, not a vendor score
from pathlib import Path
import json
import subprocess

FIXTURE = Path("fixtures/app.py")
RECEIPT = Path("env/receipt.json")
PYTEST = ["python3", "-m", "pytest", "-q", "tests/"]

def test_fixture_is_frozen():
    assert FIXTURE.is_file(), "freeze a fixture before the agent runs"

def test_receipt_has_pins():
    data = json.loads(RECEIPT.read_text())
    assert data.get("git_head"), "pin the commit under test"
    assert data.get("fixture_hash"), "pin the fixture bytes"

def test_assertions_exit_zero():
    result = subprocess.run(PYTEST, capture_output=True, text=True)
    Path("artifacts/pytest.out").write_text(result.stdout + result.stderr)
    assert result.returncode == 0, result.stdout[-2000:]
Enter fullscreen mode Exit fullscreen mode

Notice the absences. No spinner screenshot. No LGTM match against chat. No model identifier pretending to be an oracle.

If pytest never ran, you admired a patch. You did not evaluate it. Would you merge on applause?

Myth 5: Pasting the same prompt is a regression test

Claim: I sent the prompt again. Still looked fine. Regressions are covered.

Evidence you actually hold: Another sample from a moving system. Tools changed. Context changed. You never set a seed.

Corrected model: Regression means pinned artifacts plus deterministic checks. Same vibes are not a suite.

Pin these five things, or call the work exploration:

  1. Fixture files and their hashes
  2. Assertion commands and expected exit codes
  3. Dependency lockfiles
  4. The git commit under test
  5. A written note that the model is not the oracle 
# proposal: fail the job when pins are missing
set -euo pipefail
test -f tests/test_eval.py
test -n "$(git rev-parse HEAD)"
test -f requirements.txt -o -f poetry.lock -o -f package-lock.json
test -f env/receipt.txt
Enter fullscreen mode Exit fullscreen mode

Exploration is healthy. Naming exploration a regression suite is the myth. Replay bytes, not memories.

A one-hour workflow that produces artifacts

Use any scratch machine. A free server is acceptable for a draft. Do not publish that draft as a lab report.

  1. Freeze one fixture file and hash it.
  2. Write three assertions: exists, exit code, one output check.
  3. Capture the environment receipt first.
  4. Let the agent edit a working copy, never the fixture.
  5. Run assertions with no human in the loop.
  6. Store stdout, exit code, receipt, and git diff.
  7. Share those four artifacts. Leave the chat in the appendix.

Want a still smaller loop? Hash, assert, receipt, diff. Stop there.

# proposal: one working-copy rule
cp fixtures/app.py work/app.py
# agent may edit work/app.py only
sha256sum fixtures/app.py > env/fixture.sha256
git diff -- work/app.py > artifacts/work.diff
python3 -m pytest -q tests/
echo $? > artifacts/pytest.exit
Enter fullscreen mode Exit fullscreen mode

Decision table

You collected You may say You may not say
Chat trace only we tried a prompt pass rate, quality, SLO
Trace plus one pytest, no pins a check ran once reproducible eval
Pins, assertions, and receipt this commit met these checks model ranking
Same checks on owned CI the pipeline agrees that free compute is permanent

Read the right-hand column twice. That is where most Slack screenshots go to die.

Limitations

This workflow does not rank models. It does not prove production latency. It does not replace a security review.

Free model access can throttle or vanish. I am not claiming quotas, hardware, duration, or uptime. I did not publish numbers I did not measure.

The harness is a proposal. Fill real hashes before you trust it. Short sentences keep me from smuggling extra claims.

Who should skip this

Do not use this if you need certified evaluations. Do not use this for safety-critical releases. Do not use this if your team already owns a pinned eval cluster.

Skip it if you wanted a catalog of model names. I will not invent them. Skip it if you only needed a Slack screenshot. This process will feel slow. That slowness is the point.

What I will accept as an eval

I will accept fixtures, assertions, and receipts. I will not accept vibes.

Draft the loop on a free model box if that helps you start. Export the artifacts. Replay the same assertions on a machine you control.

If you want a sanity check, send the receipt schema. Do not send the chat log.

Top comments (0)