- Book: Observability for LLM Applications
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You asked for a full function and got three lines, then a comment telling you to fill in the rest yourself. In late 2023 a lot of developers got that reply from GPT-4, and they said so loudly enough that on December 8 the official ChatGPT account answered that it had heard the feedback about GPT-4 "getting lazier." The same post said the model had not been updated since November 11 and that the change was not intentional.
So the model hadn't been updated, and people were still sure it had got worse. Three days later someone offered an explanation: the model knew it was December.
By the end you'll have a fixed task set that runs against your coding agent on a schedule, so the next time it feels lazier you have numbers to check.
The complaint, and OpenAI's odd answer
OpenAI's follow-ups made things stranger. As CIO Dive reported, the company said differences in behavior can be subtle, with only a subset of prompts degraded, which makes them hard to spot and fix. It also said "training chat models is not a clean industrial process" and that "different training runs even using the same datasets can produce models that are noticeably different."
Read those together and you get an uncomfortable position for anyone shipping on top of the API. The vendor says nothing changed on its side since November 11. Users say the output changed. Both can be true if the complaints are about prompts the vendor never tested. They can also both be true if the model had been quietly lazy on some tasks all along, and only got noticed when enough people hit it.
The winter-break hypothesis goes viral
The theory going around: ChatGPT's system prompt includes the current date, and the model learned from human text that people get less done in December. So it does less too.
On December 11, Rob Lynch posted a test. He sent gpt-4-turbo the same coding prompt through the API, 477 times with a system prompt saying it was May and 477 times saying it was December. The May runs averaged a completion length of 4,298, the December runs 4,086, and a t-test gave p < 2.28e-07 (Simon Willison's note has the same numbers). He shared the code and asked people to try it.
Ethan Mollick passed it on the same day as "a statistically significant test" that the hypothesis "may be true."
The winter-break hypothesis: the model reads the date and slows down for the holidays.
Look at the size of the effect before the p-value. 4,086 against 4,298 is about 5% shorter. A tiny p-value with a 5% difference means the difference is probably not zero. It does not mean the difference is the thing users were complaining about. Nobody notices a reply that is 5% shorter. People were complaining about functions that stopped halfway.
Then the replication came in
The next day AI researcher Ian Arawjo reported that he still could not reproduce the effect at N=240. He also pointed at the method: the response lengths were not normally distributed (Shapiro-Wilk, p < 0.05), so a t-test on the means was the wrong tool. His conclusion was that ChatGPT has no "seasonal affective disorder." GIGAZINE's write-up covers the back and forth.
One run of one prompt, with a result that fits a good story, spreads faster than the dull replication that follows it.
The laziness itself was a different matter. It was real enough to measure. Aider, the open-source coding agent, had built a laziness benchmark: 89 Python refactoring tasks taken from 9 popular open-source repos and picked because they "tend to make GPT-4 Turbo write lazy comments." Each task has a test that checks the new function has about the same number of AST nodes as the original, so code replaced with a comment fails. Aider also counted new comments containing "…", the "…add logic here…" kind. GPT-4 Turbo (gpt-4-1106-preview) scored 20% with lazy comments on 12 tasks. Switching aider's edit format to unified diffs raised it to 61%, with lazy comments on 4.
Then, on January 25, 2024, OpenAI released gpt-4-0125-preview and said it "completes tasks like code generation more thoroughly than the previous preview model" and "is intended to reduce cases of 'laziness' where the model doesn't complete a task" (Decrypt, The Decoder). Aider ran its benchmark on the new model and found it scored worse with the unified-diff format. With search/replace blocks it beat the November model, but still trailed both models on unified diffs.
So the users were right that something was off. The viral explanation was wrong. And whether the vendor's fix worked depended on the edit format aider measured it with.
2023 wasn't the first time
This pattern is older than December 2023. In July 2023 a paper by Lingjiao Chen, Matei Zaharia and James Zou reported GPT-4's behavior changing between its March and June versions. Arvind Narayanan and Sayash Kapoor replied that the first version tested prime-number questions using only primes, and that its code test "merely" checked whether the output was "directly executable," so extra explanation counted as a failure. Their main point holds for every complaint since: "A model that has a capability may or may not display that capability in response to a particular prompt." Behavior can drift a lot while capability barely moves.
Sometimes the vibes are right and the cause is not the model at all. In September 2025 Anthropic published a postmortem of three infrastructure bugs that lowered Claude's response quality for weeks: a context-window routing error that at its worst hour hit 16% of Sonnet 4 requests, output corruption, and a TPU compiler bug. Anthropic wrote that it "never" reduces model quality "due to demand, time of day, or server load," and admitted its own evaluations did not "capture the degradation users were reporting." It knew reports were rising online but "lacked a clear way to connect these to each of our recent changes."
If the vendor can't connect the reports to a cause, your gut can't either. What you can do is hold everything fixed except time, and count.
A harness that tells you when your agent changed
The idea is the opposite of Lynch's test. Don't change one variable and hunt for an effect. Freeze everything you control: the tasks, the prompts, the model id, and the fixture repos. Record the agent's version on every run. Then run the same set every week and watch the counts.
Each task is a folder: repo/ (a small fixture project), prompt.txt, and check.sh (exits 0 when the work is right, for example by running the fixture's tests). Use tasks that look like your real work, and include a few that pull toward lazy output: long refactors, "rewrite this whole file," a function with many branches.
This version drives Claude Code in non-interactive mode. --bare skips your hooks, skills, plugins and CLAUDE.md, so every run starts from the same place. Bare mode needs ANTHROPIC_API_KEY set, and --output-format json gives you result and total_cost_usd (a client-side estimate). Swap the command for any agent with a headless mode.
#!/usr/bin/env python3
"""Run a fixed task set against a coding agent."""
import datetime, json, os, re
import shutil, subprocess, tempfile
from pathlib import Path
TASKS = Path("tasks")
RUNS = int(os.environ.get("RUNS", "5"))
MODEL = os.environ["AGENT_MODEL"] # a full, pinned model id
LAZY = re.compile(
r"rest of (the )?(implementation|code)"
r"|\.\.\. ?(existing|original|remaining)",
re.IGNORECASE,
)
def agent_version():
out = subprocess.run(["claude", "--version"],
capture_output=True, text=True)
return out.stdout.strip()
def run_agent(work, prompt):
cmd = ["claude", "--bare", "-p", prompt,
"--model", MODEL,
"--permission-mode", "acceptEdits",
"--output-format", "json"]
try:
out = subprocess.run(cmd, cwd=work, text=True,
capture_output=True, timeout=900)
return json.loads(out.stdout)
except (subprocess.TimeoutExpired, json.JSONDecodeError):
return {}
def lazy_hits(work):
return sum(len(LAZY.findall(f.read_text(errors="ignore")))
for f in Path(work).rglob("*.py"))
def run_task(task, stamp, version):
prompt = (task / "prompt.txt").read_text()
check = (task / "check.sh").resolve()
with tempfile.TemporaryDirectory() as tmp:
work = Path(tmp) / "repo"
shutil.copytree(task / "repo", work)
base = lazy_hits(work) # stubs already in the fixture
res = run_agent(work, prompt)
ok = subprocess.run(["sh", str(check)], cwd=work,
capture_output=True)
return {"date": stamp, "task": task.name,
"agent": version, "model": MODEL,
"passed": ok.returncode == 0,
"lazy": lazy_hits(work) - base,
"cost": res.get("total_cost_usd"),
"chars": len(res.get("result", ""))}
def main():
stamp = datetime.date.today().isoformat()
version = agent_version()
tasks = sorted(p for p in TASKS.iterdir() if p.is_dir())
with open("results.jsonl", "a") as log:
for task in tasks:
for _ in range(RUNS):
row = run_task(task, stamp, version)
log.write(json.dumps(row) + "\n")
if __name__ == "__main__":
main()
The lazy regex scans only .py files and only catches the obvious stubs. Edit it for your languages. The fixture's own tests in check.sh are the real signal, as in aider's AST-size check.
Run it from cron once a week. Cron starts with an almost empty environment and a short PATH, so put the variables and the path to claude in a small wrapper, with the API key read from a file rather than pasted in:
#!/bin/sh
# ~/code/agent-evals/weekly.sh
export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"
export AGENT_MODEL="<your-pinned-model-id>"
export ANTHROPIC_API_KEY="$(cat ~/.config/agent-evals/key)"
cd ~/code/agent-evals && python3 run.py
0 6 * * 1 sh ~/code/agent-evals/weekly.sh >> ~/code/agent-evals/run.log 2>&1
Then read the counts per run:
#!/usr/bin/env python3
"""Pass and lazy counts per run date and agent version."""
import json
from collections import defaultdict
runs = defaultdict(lambda: [0, 0, 0])
with open("results.jsonl") as f:
for line in f:
r = json.loads(line)
c = runs[(r["date"], r["agent"])]
c[0] += r["passed"]
c[1] += 1
c[2] += r["lazy"] > 0
for (date, agent), (ok, n, lazy) in sorted(runs.items()):
print(f"{date} {agent:<24} pass {ok}/{n} lazy {lazy}/{n}")
Two lessons from December 2023 apply when you read it. First, compare counts, not average lengths. Pass/fail is what you care about, and it sidesteps the distribution problem Arawjo raised. Second, know your noise. With 20 tasks at RUNS=5, that's 100 attempts per run. At a pass rate near 77%, the standard error of the difference between two runs is about 6 points, so a drop from 80 to 74 is noise. A drop you see two weeks running, or one that lines up with a new agent version in the agent column, is worth your afternoon.
When the numbers move, look at the turns
The harness tells you that something changed. It doesn't tell you what the agent did. For that you want to rerun the failing task by hand and watch it. I build Hermes IDE, so weigh this with that in mind. It doesn't schedule anything, so cron still runs the harness. Hermes is a place to do the rerun.
Press ⌘N (Ctrl+Shift+N on Windows and Linux), paste the task's prompt, and start it once in Claude Code and once in Codex. Each task gets its own hermes/ branch and worktree, so the two runs can't touch each other's files, and split panes put them side by side. The sidebar shows each session's status, context use, spend and branch. Spend is the cost the agent reports, or an estimate marked as estimated.
Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.
Every agent turn is saved, so ⌘G (Ctrl+Shift+G) opens the Review Desk and shows the changes turn by turn. If the agent wrote the real function in turn 1 and swapped half of it for a "rest of implementation" comment in turn 2, that's the turn you'll see it in. You can restore any single turn.
The Review Desk by turn: two turns of a Claude Code task, turn 1's diff of src/db.ts, and a commit message drafted from the turns.
Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. Download it or read the code on GitHub. It isn't affiliated with Anthropic, OpenAI or any agent vendor. If you only use one agent, its own CLI plus the harness above already gets you most of the way.
If this was useful
The harness above is a small, local version of what a production LLM feature needs all the time: a fixed eval set, traces of what the model actually did, and cost and quality tracked over time, so "it feels worse" turns into a chart with a date on it. That is what Observability for LLM Applications covers, from tracing to evals to catching quality regressions before your users report them.
Top comments (0)