The Bug That Survived Two Other Models
I had a bug that had beaten two other models across six separate debugging sessions — a data pipeline ordering issue that only showed up when specific record types interleaved, requiring tracing across four files to actually fix. I wanted a real comparison, not a vibe check, so I wrote a script that runs the same multi-step task against multiple models and logs what actually happens.
The Test Setup
Rather than a single prompt, this simulates a realistic multi-step debugging session: the model gets the buggy code, has to investigate, propose a fix, and the script checks the fix against my actual test suite.
import subprocess
import time
from openai import OpenAI
client = OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.z.ai/api/paas/v4"
)
BUGGY_CODE = open("transform_step.py").read()
TEST_SUITE_CMD = ["pytest", "test_transform_step.py", "-v"]
def run_debug_session(model, code, extra_body=None):
messages = [
{"role": "system", "content": "You are debugging a data pipeline. Investigate the code, form a hypothesis about the bug, and propose a fix."},
{"role": "user", "content": f"This code has an intermittent ordering bug. Find and fix it:\n\n{code}"}
]
start = time.time()
response = client.chat.completions.create(
model=model,
messages=messages,
extra_body=extra_body or {},
)
elapsed = time.time() - start
usage = response.usage
proposed_fix = response.choices[0].message.content
return {
"model": model,
"elapsed_seconds": round(elapsed, 2),
"total_tokens": usage.total_tokens,
"proposed_fix": proposed_fix,
}
def apply_and_test(fix_code, filepath="transform_step.py"):
with open(filepath, "w") as f:
f.write(fix_code)
result = subprocess.run(TEST_SUITE_CMD, capture_output=True, text=True)
return result.returncode == 0, result.stdout
results = []
for model in ["glm-5.3", "qwen3-235b-a22b"]:
session = run_debug_session(
model, BUGGY_CODE,
extra_body={"thinking": {"type": "enabled", "effort": "max"}} if "glm" in model else None
)
results.append(session)
print(f"{model}: {session['total_tokens']} tokens, {session['elapsed_seconds']}s")
I extracted just the code from each model's response and ran it against my actual test suite manually — automating that extraction reliably enough to trust unattended was more work than the comparison itself needed for a one-off test.
What Actually Happened
Both models produced plausible-looking fixes on the first pass. Only one passed my full test suite without introducing a new failure elsewhere. GLM-5.3 used meaningfully fewer total tokens to arrive at a fix that held — this matters less for a one-off debugging session and much more if you're running this kind of task repeatedly in an automated agent loop, where token cost compounds with every iteration.
The gap wasn't universal, though. I ran the same script against three smaller, single-function bugs I keep as a regression set, and the token and success-rate difference between models mostly disappeared — both handled short, self-contained fixes about equally well. The advantage showed up specifically on the task that required holding context across multiple files and forming a multi-step hypothesis, not on shorter, isolated problems.
Why This Tracks With What's Been Published
GLM's recent model releases have concentrated post-training specifically on long-horizon agentic coding — exploring a codebase, forming a plan, executing across steps, verifying the result — rather than broad general-purpose gains. Z.ai's own benchmarks show GLM-5.3 reaching higher accuracy on long-horizon coding tasks while using fewer output tokens per task than its predecessor at equivalent effort levels. That's consistent with what I saw on my specific bug, though a sample size of one real bug and three regression-set bugs is nowhere near enough to generalize from — treat this as a reason to run your own test, not as a substitute for one.
If You Want to Run This Comparison Yourself
Use a real bug from your own codebase, not a synthetic one — synthetic bugs tend to have cleaner failure signatures than the kind that actually costs you a debugging afternoon
Track both token usage and pass/fail against your actual test suite, not just "does the output look reasonable"
Test at least one short, single-step task alongside your long multi-step one — the difference between models may only show up on one type
Testing Across Models Without Rebuilding Each Time
Once I had this working against GLM directly, I wanted to add more models to the comparison without maintaining separate auth and request setups for each. I ran the same script through RouteAI for the multi-model version — same request structure, different base_url and model name per test. That's a convenience layer for running the comparison, not part of the actual result; the numbers above came from testing GLM's own endpoint directly.
TL;DR: On a real multi-file bug that had beaten two other models across six sessions, a GLM AI model produced a fix that passed my full test suite using fewer tokens than a comparison model — but the advantage was specific to the long, multi-step task, not a short single-function bug tested alongside it. Full test script above; run it against your own bug before trusting any general claim about which model is "better."
Worth exploring if this is relevant to your stack: www.fastrouteai.com
Top comments (1)
Your approach to benchmarking the models against real-world multi-file debugging scenarios is insightful! It highlights the nuances of model performance that often get overshadowed by standard metrics. It might be interesting to explore how varying the complexity of the bugs influences the models' effectiveness, perhaps by adding more intricate cases to your regression set. If you decide to expand this project or need additional development support, I’d be glad to discuss a paid collaboration! What were the most surprising results you encountered during your testing?