A Failure Triage Script for Free Coding Models: Prompt, Model, or Wrapper?

When a free coding model gives you a broken patch, you can lose an hour deciding what to fix. Earlier articles in this series covered repeatable baselines for new model releases and smoke tests for demo prompts. This one starts from the opposite end: a single failure has already happened, and you need to assign it to the right layer before changing anything.

The failure tax is real because free coding models fail in different ways. One model returns a plausible-looking stub, another omits the file you never pasted, and a third creates code that would work if only the SDK version were different. If you debug the model when the prompt was vague, or tweak the prompt when the import path is wrong, you burn cycles without learning anything portable.

A simple triage taxonomy

I treat each failed response as belonging to one of five layers:

1. Prompt shape - the instruction was too open, or the model guessed at the expected output.
2. Context gap - the model did not have the relevant file, symbol, or constraint.
3. API surface - the generated code uses an import, function, or parameter that does not exist in the target version.
4. Runtime environment - the error is about missing modules, paths, permissions, or commands, not about the generated logic.
5. Model capability - the response is off-task, hallucinated, or too weak for the subtask.

This is intentionally coarse. The point is to make the first debugging move cheap and repeatable, not to build a classifier with academic precision.

Reproducible artifact: triage.py

The script below is a starting point, not a benchmark. It receives a JSON transcript with the prompt, response, expected file, language, and optional error log. It then scores common failure signatures and prints a recommended next step.

#!/usr/bin/env python3
"""Triage one failing coding-model response.

Feed a JSON transcript with:
  prompt, response, expected_file, language, error_log
The script scores common failure signatures and prints a recommended layer.
"""
import json, re, sys
from pathlib import Path


def signature_hits(response, error_log, language):
    return {
        "instruction_shape": [
            bool(re.search(r"\bTODO\b|\bplaceholder\b|\bexample\b", response, re.I)),
            len(response.strip()) < 60,
        ],
        "context_gap": [
            "not provided" in response.lower(),
            "missing" in response.lower(),
            "context" in response.lower(),
        ],
        "api_surface": [
            bool(re.search(r"(undefined|not exported|no module named|has no attribute)", error_log, re.I)),
        ],
        "runtime_env": [
            bool(re.search(r"(cannot find module|no such file|permission denied|command not found)", error_log, re.I)),
        ],
        "capability": [
            bool(re.search(r"\bhallucinat|\bunrelated\b|\bpretend\b", response, re.I)),
        ],
    }


def recommend(hits, transcript):
    reason = []
    if any(hits["instruction_shape"]):
        reason.append("Response reads like a stub; rewrite the prompt with an exact contract and expected output shape.")
    if any(hits["context_gap"]):
        reason.append("Response says context is missing; add only the relevant files, not the whole repo.")
    if any(hits["api_surface"]):
        reason.append("Error points at imports or APIs; verify against the target SDK version instead of retrying the model.")
    if any(hits["runtime_env"]):
        reason.append("Error is environmental; reproduce in a clean container before blaming the generated code.")
    if any(hits["capability"]):
        reason.append("Response is off-task; switch to a smaller, more explicit subtask or another model.")
    if not reason:
        reason.append("No strong signature; diff the expected file against the generated file line by line.")
    return reason


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: triage.py transcript.json")
    t = json.loads(Path(sys.argv[1]).read_text())
    hits = signature_hits(t["response"], t.get("error_log", ""), t.get("language", "unknown"))
    for layer, flags in hits.items():
        print(f"{layer:16} {sum(flags)}/{len(flags)}")
    print("\nRecommended next step:")
    for line in recommend(hits, t):
        print(f"- {line}")

A sample transcript looks like this:

{
  "prompt": "Add a retry wrapper to the client call in src/api.py",
  "response": "Here is an example implementation...",
  "expected_file": "src/api.py",
  "language": "python",
  "error_log": "NameError: name 'retry' is not defined"
}

Running triage.py on that transcript would flag both instruction_shape and api_surface. The recommendation would be to tighten the prompt and verify the retry import before changing the model or adding more context.

Where free model access and a free server help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises free model access and a free server option. For this workflow, that matters mainly because triage is iterative: you want to replay the same failing prompt across several models and against a clean environment without paying per attempt. The script above can be pointed at whatever OpenAI-compatible endpoints you have. Keep the transcript stable, change only the endpoint or the prompt edit, and the recommended layer becomes a before/after signal rather than a one-off guess.

I am not claiming MonkeyCode has unlimited rate limits, permanent model availability, or specific hardware. Treat the free tier the way you would treat any free resource: good for exploratory triage, not for a production regression suite. Run the script locally first, then move it to a server only if the provider's terms and your data allow it.

Limitations of this approach

• The signatures are heuristic. A model can produce a short, correct answer and get misclassified as a stub.
• It does not measure correctness. It only suggests which layer to inspect first.
• It requires an error log or a generated file to compare against. Purely conversational failures need different signals.
• It will not rescue a genuinely weak model; it just helps you stop wasting time on the wrong fix.

Who should skip this

Skip this script if you need deterministic API contracts, legal or data-privacy guarantees, or long-term benchmark stability. Skip it if your team already has a full evaluation harness with versioned fixtures and human review. This is a lightweight field tool for individual developers who mostly want to know: should I fix the prompt, the context, the environment, or the model?

If you maintain a small model-eval workflow, try adding this triage pass next time a free coding model fails. It usually saves more time than tweaking the prompt first.