- Book: Observability for LLM Applications
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
On September 29, a pull request merged into Codex gave GPT-6.1 Sol "the highest catalog priority" (#49318). That line made it the model the bundled catalog hands a Codex session when you don't ask for one.
Nine days later, Codex CLI 0.162.0 changed where a fresh terminal thread gets its default model from. Six days after that, GPT-5.5 left Codex for everyone signed in with ChatGPT.
Three changes in 15 days, and none of them showed up in your repo's diff. If your agent runs on whatever the default is, the thing your evals measure may have moved while your code stood still.
My take: a model you didn't choose is a dependency you didn't pin. Pin it per task, and write it down next to every result.
What changed between September 29 and October 14
Here is the sequence, from OpenAI's own release notes and docs:
| Date | Change | Who it hits |
|---|---|---|
| Sep 29 | Codex CLI 0.159.1: GPT-6.1 Sol becomes the default in the bundled catalog | CLI users who never set a model |
| Oct 7 | 0.161.0 lists it again: default in the bundled and Amazon Bedrock catalogs | The same, plus Bedrock users |
| Oct 8 | 0.162.0: new TUI threads respect the server's model defaults | TUI users who launch without a model override |
| Oct 14 | GPT-5.5 retires from Codex for ChatGPT sign-ins, at noon PDT (19:00 UTC) | Anyone who picked gpt-5.5 and signs in with ChatGPT |
The October 8 change is the quiet one. The release note says Codex will "respect server model and reasoning-summary defaults for new TUI threads while retaining explicit launch overrides." The pull request behind it (#50013) puts it more plainly: "Fresh TUI threads should use the app server's model settings unless the launch explicitly overrides them." The server here is the Codex app server the TUI talks to, a local daemon or a remote one (#50913). The same release stopped appending "(default)" to entries in the TUI model picker (#51235).
The October 14 change hits the opposite group: people who did pin a model. OpenAI's models page says the retirement covers ChatGPT sign-ins only and API-key sessions are not affected. It points Plus, Pro, Business, Enterprise and Edu users to GPT-6.1 Sol, and Free and Go users to GPT-6 Luna in the desktop app, both "when available."
To be fair to OpenAI, none of this was hidden. Every change is in a release note or the changelog, the retirement came with an exact time and a named replacement, and 0.162.0 also moved the model and reasoning effort near the top of task details (#50727). GPT-6.1 Sol may well be the better model for you. OpenAI says it nearly matches GPT-6 Astra at about a fifth of Astra's token price (Unite.AI, The Batch), though those are OpenAI's own numbers. An upgrade still moves your numbers, and your evals should say so.
Who picked the default? A catalog file
Look at what "default" meant in #49318. Nobody on your team chose GPT-6.1 Sol. It got the highest priority in a catalog file that ships inside the CLI, so it won.
Codex isn't unusual here. Claude Code's model docs say the aliases opus, sonnet and haiku "point to the recommended version for your provider and update over time." Same doc: "To pin to a specific version, use the full model name, for example claude-opus-5-5." So claude --model opus is a moving target by design, and claude --model claude-opus-5-5 is a pin.
Moving targets are fine for a chat. You want the newest model when you ask a question at 11pm. They are a problem for anything you measure, compare or promise to a teammate.
A default model can change between two eval runs without a single line of your repo changing.
Your evals measure the model, so they need its name
An agent eval can be a formal suite. More often it is a sentence in a team chat: "Codex is worse at migrations than Claude Code." "That AGENTS.md rewrite made the agent better." "This task took 40 minutes last week and 12 this week."
Each of those is a claim about a model. If the model was a default, you don't know which one it was. Did your prompt get better, or did the catalog move? Did Codex get worse at migrations, or did a fresh thread pick up a different server default than the one you tested last month?
The model is not the only hidden variable either. Reasoning effort changes results, and #50013 treats it like the model: if the launch doesn't set it, a fresh TUI thread takes the server's default. The account matters too: the GPT-5.5 retirement hit ChatGPT sign-ins and left API-key sessions alone, so the same gpt-5.5 setting meant two different things depending on how you logged in.
So the minimum record for any agent result is four fields: the model, the effort, the account type and the commit it ran against. Without them, a before-and-after comparison is two numbers with no explanation attached.
Pin it per task, then write it down
For daily work in Codex, a profile is the cleanest pin. Per the advanced config docs, a profile is a file in ~/.codex/ with top-level keys:
# ~/.codex/pinned.config.toml
model = "gpt-6.1-sol"
model_reasoning_effort = "medium"
Start it with codex --profile pinned. PR #50013 keeps explicit profile launches on the resolved configuration, so the profile's model holds even for fresh TUI threads.
For anything you compare, put the pin and the record in the same script. codex exec takes -m for the model (models page), and -c sets any config key (the config docs say a value that isn't valid TOML is treated as a string):
#!/usr/bin/env bash
# run-task.sh "<task>" : one pinned run, one logged row
set -u
task="$1"
model="gpt-6.1-sol"
effort="medium"
account="chatgpt" # or "apikey": how this run signs in
codex exec -m "$model" \
--sandbox workspace-write \
-c model_reasoning_effort="$effort" \
"$task" > last-run.log
agent=$?
npm test > test.log 2>&1
tests=$?
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$(date -u +%FT%TZ)" "$(git rev-parse --short HEAD)" \
"$model" "$effort" "$account" "$agent" "$tests" >> runs.tsv
By default codex exec runs in a read-only sandbox (non-interactive mode docs), so --sandbox workspace-write is what lets the agent edit the checkout before the tests run. The agent column keeps Codex's own exit status, so a failed agent step shows up in the row instead of hiding behind a test result on untouched code.
Swap npm test for your own check, and set account to match how the machine signs in. Every row in runs.tsv now carries all four fields: which model, effort and account produced which test result on which commit.
Pinning has a cost, and October 14 showed it. Anyone pinned to gpt-5.5 with a ChatGPT sign-in had to edit their config. OpenAI's list of what to update covers "workspace defaults, saved model settings, managed configurations, custom agents, scheduled tasks, and scripts that still select gpt-5.5" (changelog).
That is the good kind of break. You change one line in a diff, rerun your tasks, and compare the new rows against the old ones.
Where Hermes IDE fits
I build Hermes IDE, so weigh this section with that in mind. Hermes runs each agent's own CLI in a real terminal, Codex and Claude Code included, each task on its own branch and worktree.
The part that matters here: when you start a task with ⌘N (Ctrl+Shift+N on Windows and Linux), you pick the agent, model, effort and account for that task, and you can save combinations you use often as presets (README). Several accounts per agent are supported, so the sign-in a task runs under is something you choose when you start it.
The ⌘N New task sheet in Hermes 2.3.4: agent, project, a new hermes/ worktree, Accept edits, the model and effort chips, and a preview of the command Hermes will run.
After launch, every session shows its own spend: the cost the agent reports, or an estimate from its transcript marked as estimated, with token totals. Since 2.3.4, a Claude Code or Codex session keeps showing its own model and cost even when the conversation mentions another company's model. The bar above each session puts Model in the same place for every agent.
Codex and Claude Code side by side in Hermes 2.3.4, each in its own hermes/ worktree, with status, context use, cost and branch per session in the sidebar.
What it doesn't do: Hermes doesn't write your runs.tsv, and it can't stop a vendor from retiring a model you picked. It makes the choice explicit per task and keeps it visible per session. The record is still yours. Hermes is free, source-available under BSL 1.1, runs on macOS, Windows and Linux, and is not affiliated with OpenAI, Anthropic or any other agent vendor.
So, which model ran your last eval?
Open the last result you shared with your team, a benchmark, a "this agent is better at X" message, a before-and-after on a prompt change. Can you name the model and effort that produced it?
If not, you were testing a catalog entry. Tell me in the comments whether you pin your agent's model, or whether you'd rather ride the default and take the upgrades as they come.
If this was useful
Writing the model next to every result is the smallest piece of LLM observability. Observability for LLM Applications covers the rest of it: tracing each call, running evals that catch quality regressions when a model changes, and tracking cost per feature in production. If you run several agents, Hermes IDE is free and lets you set the model per task (GitHub).
Top comments (1)
how does promoting gpt‑6.1 sol to the top catalog slot affect sessions that were still targeting gpt‑5.5, especially around fallback handling?