Six Harness Anti-Patterns That Fake Model Failure

Stop blaming the model for a dirty harness. Six request-shaping habits waste free inference and hide real bugs. I name each habit, then I replace it.

The catalog is for coding agents on cheap endpoints. It is not a vendor scorecard. If your traces look clean, skip the product talk entirely.

What I mean by harness

The harness is everything around the weights. Tools, retries, logs, and context packing all count. The model is the last suspect, not the first.

You rewrote the prompt three times last night. Did the request shape change at all? If not, you only shuffled adjectives.

How to read each item

Every anti-pattern uses the same four beats. Symptoms. Root cause. Replacement. A cheap check. Keep a JSONL request log or this catalog cannot help you.

I am not claiming a published benchmark here. I am claiming a repeatable inspection habit.

1. Whole-Repo Paste

Symptoms

• One user message dwarfs the failing test.
• The model restates files you never named.
• Latency climbs while the diff stays noisy.

Root cause

You treat context like a dump truck. Retrieval never happened. Neighbors of the bug drown the invariant.

Replacement: slice, then cite

Send the failing test, the target file, and the contract. Cite paths in the prompt. Leave the rest on disk.

# allowed pack for one coding turn
- tests/test_invoice.py   # failing example
- invoice/totals.py       # target
- invoice/types.py        # types only

Check

Cap a single coding turn with a character budget. Start at twelve thousand characters, then tune. If you need the whole repo, you need search, not a bigger paste.

Why did the agent need node_modules in chat?

2. Happy-Path Tool Schema

Symptoms

• Tool payloads look successful with empty bodies.
• The agent repeats the same call without new evidence.
• Chat says "done" while the test still fails.

Root cause

Your schema has ok and stdout. It has no error_class. Failure looks like silence, so the model invents a story.

Replacement: errors as data

from typing import Literal, TypedDict

ErrorClass = Literal["", "timeout", "not_found", "assert", "schema"]

class ToolResult(TypedDict):
    ok: bool
    error_class: ErrorClass
    stdout: str
    stderr: str
    elapsed_ms: int

Return structured failure from every tool. Do not let the model guess why a subprocess went quiet.

Check

Grep traces for tool messages with empty stderr and ok: true. If tests failed in that turn, your schema lied.

3. Session That Never Dies

Symptoms

• Turn fourteen still mentions a deleted file.
• The model "remembers" an API you reverted.
• Token use grows while the task shrinks.

Root cause

You never reset the thread. Stale assistant text becomes fake source. Cheap inference makes that habit feel free. It is not free in correctness.

Replacement: ephemeral tasks, durable facts

New bug, new session. Persist the spec file and the failing test. Drop the chat history on purpose.

# proposed reset, labeled as a workflow
cp task.md /tmp/task.md
: > session.jsonl

Why is a week-old thread still the workspace?

Check

Flag any session past eight model turns without a new failing test. That bound is a starting heuristic, not physics.

4. Spec Written After the Diff

Symptoms

• The PR body is a chat export.
• Reviewers argue about intent, not code.
• You cannot replay the task next month.

Root cause

Chat felt faster than a contract. The contract still exists. It is just scattered across bubbles.

Replacement: spec file first

# task.md
## invariant
`totals()` never returns negative cents.
## fail
pytest tests/test_invoice.py::test_refund_rounding
## done
That test passes. No other test is deleted.

The model may edit code. The spec does not live in scrollback. If the spec changes, that is a new task.

Check

Refuse to start an agent turn without task.md in the pack. No file, no request.

5. Retry Until the Tests Blur

Symptoms

• Five regenerations, same stack trace.
• You cannot name the delta between tries.
• Someone says the free model is just weak.

Root cause

Retry is not a search strategy. You mutated hope, not evidence. The harness hid that you asked the same question.

Replacement: one change per retry

Change only one axis: context slice, tool result, or instruction. Log the delta in a file. Stop after a hard bound.

# proposed loop, not a magic fixer
for i in 1 2 3; do
  python harness_lint.py --log last.jsonl || exit 1
  echo "attempt=$i" >> delta.md
  run_agent --attempt "$i" --spec task.md
done

Check

If two adjacent requests hash to the same pack, you did not retry. You refreshed.

6. Shared Endpoint, Shared Secrets

Symptoms

• .env fragments appear in prompts.
• Every toy agent points at one public URL.
• Nobody can say who stored the last trace.

Root cause

Free inference feels like a scratch pad. Scratch pads leak. A shared server is not your staging cluster.

Replacement: redact, isolate, then call

Strip secrets before the request leaves the machine. Use a throwaway project for public endpoints. Keep production traffic off the shared box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode only because its free model access and free server option are a convenient place to re-run a cleaned request, not a place to paste production env files. The linter below still matters if you never touch that product.

Check

Scan the JSONL for token-shaped strings before you send anything. Fail the turn if one matches.

Artifact: lint the request log

This is a small, local checker. Point it at JSONL traces you already store. It does not call a model. Label it as a harness test, not a quality score.

Fixture: sample.jsonl

{"turn":1,"chars":18000,"tools":[{"ok":true,"stderr":"","stdout":""}],"has_spec":false,"session_turns":12,"same_pack_as_prev":true,"prompt":"API_KEY=sk-demo-not-real\nplease fix invoice totals"}

Checker: harness_lint.py

#!/usr/bin/env python3
"""Flag request-shaping anti-patterns in JSONL agent traces."""
from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

SECRET = re.compile(r"(api[_-]?key|secret|token)\s*=\s*\S+", re.I)
CHAR_BUDGET = 12_000
SESSION_BUDGET = 8


def findings(row: dict) -> list[str]:
    out: list[str] = []
    if int(row.get("chars", 0)) > CHAR_BUDGET:
        out.append("whole_repo_paste")
    tools = row.get("tools") or []
    for tool in tools:
        if tool.get("ok") and not tool.get("stderr") and not tool.get("stdout"):
            out.append("happy_path_tool_schema")
            break
    if int(row.get("session_turns", 0)) > SESSION_BUDGET:
        out.append("session_that_never_dies")
    if not row.get("has_spec"):
        out.append("spec_written_after_diff")
    if row.get("same_pack_as_prev"):
        out.append("retry_until_tests_blur")
    text = str(row.get("prompt", ""))
    if SECRET.search(text):
        out.append("shared_endpoint_shared_secrets")
    return out


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--log", required=True)
    args = parser.parse_args()
    path = Path(args.log)
    failed = 0
    for line_no, line in enumerate(path.read_text().splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        hits = findings(row)
        if not hits:
            continue
        failed += 1
        print(f"line {line_no}: {', '.join(hits)}")
    print(f"flagged_turns={failed}")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Command

python harness_lint.py --log sample.jsonl

Expected output

line 1: whole_repo_paste, happy_path_tool_schema, session_that_never_dies, spec_written_after_diff, retry_until_tests_blur, shared_endpoint_shared_secrets
flagged_turns=1

If that fixture does not fail, the script drifted. Fix the checker before you lecture the model.

Map each trace field on purpose. chars is packed prompt size. has_spec is whether task.md was attached. same_pack_as_prev is your own hash compare. Do not invent those fields after the run.

Decision table

Signal in the log	Do not do this	Do this instead
Prompt larger than the failing test	Paste the repo	Pack test + target + types
Tool ok with empty streams	Ask the model why	Return error_class
Session longer than the bug	Keep chatting	Reset, keep task.md
PR description is a transcript	Merge on vibes	Write invariants first
Adjacent requests are identical	Hit regenerate	Change one axis, log it
Secrets in the prompt	Use a shared demo box	Redact, then consider any free server

What this does not prove

The linter does not grade model quality. It grades your request shape. False positives happen on generated fixtures and on legitimately large generated files.

It is not a security scanner. The secret regex is a tripwire. Rotate anything that already left your disk.

I also do not claim these bounds are universal. Twelve thousand characters and eight turns are starting knobs. Publish your own knobs if your stack disagrees.

Who should skip this

Skip it if you have no request logs. You cannot lint a vibe.

Skip it if you need a contractual SLA from a shared free endpoint. This workflow assumes throwaway tasks and local tests.

Skip it if your agent already ships with retrieval, structured tool errors, and a spec file. You are not the audience. You already left the anti-patterns.

Do not point a dirty harness at any free server and call that an evaluation. You will measure paste quality, not model quality.

The move I want

Run the fixture first. Then point the same script at one real trace. Fix the highest finding before you switch models.

If you already have a clean pack and still want a second cheap run, MonkeyCode's free server option is one place to try that cleaned request. Keep secrets out of it either way.