DEV Community

Cover image for Why a Local Model Looked Broken and What Actually Capped It
4sapi
4sapi

Posted on

Why a Local Model Looked Broken and What Actually Capped It

A local Qwen build with a 131,072-token context refused to emit more than 3,208 tokens per response. The server was fine. The model was fine. The cap lived in a client-side fallback record that nobody had written. Here is the full diagnostic chain, the metadata schema that fixes it, and the 401 trap that sits one step to the side.


1. The symptom, in numbers

The setup: codex CLI pointed at a local Qwen3.8-Flash-Next IQ3_S build served on 127.0.0.1:8080, on a machine with an RTX 5060 Ti 16 GB, 64 GB DDR4-3600, and a declared context window of 131,072 tokens.

Normal question answering worked. Small file edits worked. Then a script of a few hundred lines was handed over for modification, and the run stopped mid-sentence with two lines on screen:

writing a tool call: 2368 of max 3208 tokens, 61.6 tok/s
Incomplete response returned, reason: max_output_tokens
Enter fullscreen mode Exit fullscreen mode

The arithmetic is the whole story. 3,208 of 131,072 is 2.4%. The model was allocated less than one fortieth of its own context window as a single-response budget. A multi-hundred-line rewrite cannot fit in 2.4% of the available space, so the run terminates by design, not by failure.

Two numbers in that log line are worth separating. 2368 of max 3208 is progress against a hard ceiling. 61.6 tok/s is throughput, and it matters later: it sets the cost of raising the ceiling.

The tell appears at startup, before any task runs:

warning: Model metadata for `your-model-name` not found.
Defaulting to fallback metadata; this can degrade performance and cause issues.
Enter fullscreen mode Exit fullscreen mode

That warning is the entire diagnosis. The client does not recognize the model, so it applies a built-in generic record. 3,208 is that record's number, not the model's.


2. Before editing anything: decide whether you need to

Most people reach for configuration at this point. The cheaper question comes first: does this situation actually require a config change?

Your situation What you see What to do
Official hosted models (GPT-5 / Claude family) No metadata warning Nothing. This article does not apply
Local model, short Q&A and small edits only Occasional truncation Optional. Split the task instead (Option A)
Local model, long files or long patches Repeated Incomplete response Fix the catalog (Option C)

One test decides it. If codex prints Model metadata for ... not found at startup, your model is unregistered and the fallback is active. If that line is absent, the cap has a different origin and this whole procedure is wasted effort.


3. Three fixes, ranked by cost

Before touching a configuration file, establish which fix you actually want.

Option A — shrink the task. One file per request, no more than roughly 200 lines at a time, split large files across several runs. Zero cost, but it breaks the workflow: each split loses the thread of the change.

Option B — change the client. The same server, a different front end. Claude Code talks to /v1/messages; a direct connection to port 8080 carries no 3,208 cap. This suits teams that already switch between clients and want to keep both.

Option C — supply the missing metadata. Teach the client what this model is, once, and every subsequent task benefits. That is the route documented below.

If codex is the long-term tool, take C. If the goal is to finish today's work, take A.


4. Confirm which side owns the bottleneck

Skipping this step is the most common cause of fixing the wrong file. One request separates server from client:

curl -s -X POST http://127.0.0.1:8080/v1/messages \
  -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"x","max_tokens":8192,"messages":[{"role":"user","content":"print integers 1 to 2000, one per line, nothing else"}]}'
Enter fullscreen mode Exit fullscreen mode

Two fields in the response decide the case:

stop_reason: max_tokens
output_tokens: 8192
Enter fullscreen mode Exit fullscreen mode

output_tokens: 8192 proves the server will emit at least 8,192 tokens when asked. The ceiling therefore sits in the client. Repeated with max_tokens at 100,000, 32,768, and 16,384, the server accepted every value — no error, no truncation. The server side is clean.

The probe is also portable. The identical request works against a hosted endpoint behind a gateway such as 4sapi, which makes it a reusable check for any workflow that mixes local and cloud backends: one request, one output_tokens field, and the ceiling localizes to a layer.

If your server errors or clips at this stage, stop. The problem is upstream, in the server's own max_tokens handling, and nothing in the client configuration will help.


5. The fast path: one script, two writes

The script performs two operations and nothing else: generate model-catalog.json, then point config.toml at it. Saved as 生成-model-catalog.ps1, it takes four parameters.

Parameter Value used Purpose
-Spec `slug\ display name\
{% raw %}-ContextWindow 131072 Writes max_context_window to match the server
-AutoCompact 100000 History compression threshold
-CatalogPath / -ConfigPath / -DryRun defaults Override paths, or preview without writing

Internally it builds one entry per spec row, serializes the models array to JSON at depth 10, writes UTF-8 without BOM, then edits config.toml. The edit is line-based: it appends model_catalog_json = "<path with escaped backslashes>" if the key is missing, and if model_auto_compact_token_limit is present it rewrites that value in place. Before writing, it copies config.toml to config.toml.bak-YYYYMMDDHHmmss.

Run it with your model's identifier:

.\生成-model-catalog.ps1 -Spec 'Qwen3.8-flash-next-iq3_s|Qwen3.8 IQ3_S (local)|16384'
Enter fullscreen mode Exit fullscreen mode

Where the identifier comes from: either the startup warning (Model metadata for 'xxx' not found) or a direct query of curl http://127.0.0.1:8080/v1/models and reading the id field.

The script is not required. If you want to know what each field does before trusting it, the manual path follows.


6. The one field that decides output length

Everything else is compliance. One field sets the ceiling:

"truncation_policy": { "mode": "tokens", "limit": 16384 }
Enter fullscreen mode Exit fullscreen mode

mode accepts bytes or tokens. With tokens, limit is the per-response maximum.

Two naming traps are worth stating explicitly, because both cost real time.

The field is not max_output_tokens. It is not model_max_output_tokens either — that key existed in earlier builds and was retired in codex 0.169. Writing it produces a startup notice and nothing else:

user (C:\Users\zsl\.codex\config.toml): `model_max_output_tokens` is ignored
Enter fullscreen mode Exit fullscreen mode

The configuration is accepted, the warning is not fatal, and the cap does not move. That is the worst failure mode in this whole procedure: a change that looks applied.


7. The catalog entry, field by field

A complete minimal entry carries 29 keys. Several are structural rather than functional, which is why hand-writing from memory produces the error cascade in section 9.

{
  "models": [
    {
      "slug": "qwen3.8-flash-next-iq3_s",
      "display_name": "Qwen3.8 IQ3_S (local)",
      "default_reasoning_level": "none",
      "supported_reasoning_levels": [{ "effort": "none", "description": "no reasoning" }],
      "shell_type": "default",
      "visibility": "list",
      "supported_in_api": true,
      "priority": 1,
      "additional_speed_tiers": [],
      "service_tiers": [],
      "available_access_programs": { "cyber": [] },
      "model_messages": {},
      "base_instructions": "You are a helpful coding assistant running locally.",
      "include_skills_usage_instructions": false,
      "include_plugin_usage_instructions": false,
      "include_apps_usage_instructions": false,
      "supports_reasoning_summary_parameter": false,
      "support_verbosity": false,
      "truncation_policy": { "mode": "tokens", "limit": 16384 },
      "supports_image_detail_original": false,
      "max_context_window": 131072,
      "auto_compact_token_limit": 100000,
      "effective_context_window_percent": 100,
      "experimental_supported_tools": [],
      "input_modalities": ["text"],
      "supports_search_tool": false,
      "supports_experimental_context": false,
      "use_responses_lite": false,
      "supports_reasoning_effort_updates": false
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Save it as C:\Users\<you>\.codex\model-catalog.json.

The functional subset is smaller than the schema suggests:

Field Role Typical value
slug Must match the model id in the client's request exactly; a miss returns the fallback silently your model id
display_name Label in the UI model list free text
truncation_policy Single-response ceiling {mode:"tokens", limit:16384}
max_context_window Advertised context; must match the server 131072
auto_compact_token_limit When history compression triggers ~75% of context
base_instructions System prompt; required one sentence
shell_type Tool execution mode default
visibility Whether the model appears in the picker list
supported_reasoning_levels Declared effort tiers [{effort:"none", description:"no reasoning"}]
available_access_programs Access allow-list; must be an object {"cyber":[]}

Then one line in C:\Users\<you>\.codex\config.toml. Windows paths need doubled backslashes:

model_catalog_json = "C:\\Users\\zsl\\.codex\\model-catalog.json"
Enter fullscreen mode Exit fullscreen mode

The key name contains "json" but the value is a path. Pasting the JSON content in instead produces:

Error: 文件名、目录名或卷标语法不正确 (os error 123)
Enter fullscreen mode Exit fullscreen mode

A file-path error message, caused by a type error. Worth recognizing on sight.


8. What actually happened, mechanically

Three facts combine into the observed behavior.

  1. The client resolves a model against its catalog at request build time. Unknown slug, no entry.
  2. No entry, a built-in fallback record is substituted, and truncation_policy in that record is {mode: tokens, limit: 3208}.
  3. The server is asked for at most 3,208 output tokens, honors that, and returns stop_reason: max_tokens. The client surfaces it as Incomplete response, which reads like a model failure and is nothing of the kind.

The 131,072 context was never the constraint. The context window and the per-response ceiling are independent values, and only the second one truncates.


9. The error cascade, and why it is a schema contract

Hand-writing the entry produces a predictable sequence of failures. They are listed here in order of appearance, because each one hides the next: the parser stops at the first missing field, so a single bad file can take a dozen rounds to clear.

Error Cause Fix
model_max_output_tokens is ignored Deprecated key in 0.169 Delete it; use truncation_policy
os error 123 (invalid filename) JSON pasted as the path value Put the file path in model_catalog_json
missing field supported_reasoning_levels Field absent Add [{effort, description}]
expected struct ReasoningEffortPreset Array of strings passed Elements must be objects
missing field shell_type Field absent default / local / shell_command / unified_exec
missing field visibility Field absent list / hide / none
missing field supported_in_api / priority Fields absent true / 1
missing field support_verbosity Field absent false
missing field truncation_policy Field absent See section 6
missing field mode / missing field limit Sub-fields absent tokens or bytes; a number
invalid length 0, expected struct ModelAccessPrograms Wrong container shape {"cyber": []}
expected a sequence (cyber) String or boolean passed {"cyber": []}
invalid type: null, expected string or map Key written as null Delete the line; do not null it
missing both base_instructions and model_messages.instructions_template No system prompt at all Add base_instructions

Two patterns explain nearly all of them. The schema uses objects where prose suggests scalars (supported_reasoning_levels is a list of {effort, description} structs, not a string), and it uses required-with-defaults, where a missing key is an error rather than a fallback.


10. Reading the schema out of the binary

Your build may not be 0.169. Guessing field names across versions is wasteful. codex is a Rust binary, and Rust embeds serialized field names as ASCII in declaration order. They can be read straight out:

$bin = Get-ChildItem "$env:APPDATA\npm\node_modules\@openai\codex" -Recurse -Filter codex.exe |
  Select-Object -First 1
$text = [System.Text.Encoding]::ASCII.GetString([System.IO.File]::ReadAllBytes($bin.FullName))
$i = $text.IndexOf("supported_reasoning_levels")
$text.Substring([Math]::Max(0,$i-1200), 1400) -replace '[^\x20-\x7E]','|'
Enter fullscreen mode Exit fullscreen mode

Output is a run of field names in declaration order:

... slug display_name default_reasoning_level supported_reasoning_levels
shell_type visibility supported_in_api priority additional_speed_tiers
service_tiers default_service_tier available_access_programs ...
truncation_policy supports_image_detail_original max_context_window ...
Enter fullscreen mode Exit fullscreen mode

Struct arity is available the same way. Searching for struct X with N elements gives the expected field count, which turns "did I miss one?" into a check rather than a guess:

struct TruncationPolicyConfig with 2 elements
struct ReasoningEffortPreset with 2 elements
struct ModelAccessPrograms with 1 element
Enter fullscreen mode Exit fullscreen mode

This technique generalizes. Any Rust CLI with a serde schema can be inspected this way, without documentation and without a version mismatch.


11. Verification: three gates, measured

A configuration change without a test is indistinguishable from a lucky coincidence. Three gates, in ascending cost.

Gate 1 — the warning is gone.

codex --profile strata exec --skip-git-repo-check "只回复 OK"
Enter fullscreen mode Exit fullscreen mode

No Model metadata ... not found line means the catalog was parsed. A second, more visible confirmation: the session header now prints the catalog's display_name.

before: qwen3.8-flash-next-iq3_s default    ← fallback, model unrecognized
after:  Qwen3.8 IQ3_S (local) default       ← display_name from the catalog
Enter fullscreen mode Exit fullscreen mode

Gate 2 — long output is not clipped.

codex --profile strata exec --skip-git-repo-check "输出从 1 到 1500 的整数,每行一个,不要输出任何其他文字"
Enter fullscreen mode Exit fullscreen mode

Measured result: 84 seconds, no Incomplete response, largest value 1500 — complete.

Gate 3 — one step past the old ceiling.

codex --profile strata exec --skip-git-repo-check "输出从 1 到 2500 的整数,每行一个,不要输出任何其他文字"
Enter fullscreen mode Exit fullscreen mode

2,500 lines is roughly 5,000+ tokens, or 1.6× the old 3,208 limit. Measured: full output, 120 seconds.

All three passing is the bar for "fixed." Two of three usually means the slug is wrong, the file is not being read, or the limit is being overridden elsewhere.

A longer field observation sits alongside these gates: a multi-file repair session ran 34 minutes continuously after the fix, with no interruption. Hardware during that run: VRAM 15.4 / 16 GB, RAM 61.0 GB, GPU utilization 33%, CPU 52%. That profile — memory nearly saturated, compute idle — is what a local model doing real work looks like, and it sets expectations for capacity planning.


12. Raising the ceiling further

Changing truncation_policy.limit to, for example, 32768 takes effect immediately, with no restart. Two constraints govern how far this can go.

Latency is linear in output length. At a measured 61 tok/s, a 5,000-token answer costs about 120 seconds. A 32,768-token answer is roughly nine minutes of single-request wall time. With an interactive client, that reads as a hang. Raising the cap converts a truncation error into a patience problem.

Input plus output must stay within max_context_window. Raising the output limit past the context budget produces a different failure at the same boundary, not a fix.

For multiple models, add entries to the models array, each with its own slug. Two quantizations of the same model on one machine are two entries, not two files. Moving to another machine means rerunning the script with a new -Spec; config.toml does not need to change, because the script maintains that line itself.


13. Rollback

The script leaves a timestamped backup on every write:

Get-ChildItem "$env:USERPROFILE\.codex\config.toml.bak*" | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Copy-Item "<that file>" "$env:USERPROFILE\.codex\config.toml" -Force
Enter fullscreen mode Exit fullscreen mode

Deleting model-catalog.json returns the client to the fallback state on its own. Two moves, no uninstall.


14. Four misconceptions that cost the most time

Misconception 1 — "the model can't do it." The server accepted max_tokens: 100000 without complaint. 3,208 was a client-side default, not a capability boundary. Model capability and client allocation are separate questions, and mixing them sends people shopping for a new model.

Misconception 2 — "editing the profile is enough." Provider definitions and global keys live in ~/.codex/config.toml. A profile only overlays that file. Editing inside a profile leaves the base value untouched, and the change appears to do nothing.

Misconception 3 — "slug is a free label." It is a lookup key. A miss returns the fallback record, silently, without a warning beyond the startup line everyone ignores.

Misconception 4 — "larger is always better." A limit of 131,072 passes validation and works. It also turns a single response into a multi-minute request that looks like a deadlock. The useful value is the largest one your workflow can tolerate waiting for.


15. The adjacent trap: one environment variable, two readers

A second failure appeared during the same session, and it belongs here because it teaches the same lesson from the other direction.

Symptoms:

unexpected status 401 Unauthorized: missing or wrong API key,
url: http://127.0.0.1:8080/v1/responses
Enter fullscreen mode Exit fullscreen mode

The server's own startup log ended with:

ready: http://127.0.0.1:8080/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 131072 tokens, images on, API key required)
Enter fullscreen mode Exit fullscreen mode

One mechanism explains it, and it is a single variable read by two processes:

  1. The server reads STRATA_API_KEY at startup. A non-empty value switches on mandatory authentication for /v1/*.
  2. The client's provider block sets env_key = STRATA_API_KEY.
  3. The client checks only whether the variable exists, not whether its value is correct.

So "give the client a dummy key" flips the server into enforced-auth mode while the client sends nothing valid. Result: 401. The fix intent makes the failure worse.

To clear it, all three of these must happen together:

# 1. remove the user-level variable
# 2. delete the env_key line from [model_providers.*] in ~/.codex/config.toml
# 3. restart the model server so it re-reads the environment
Enter fullscreen mode Exit fullscreen mode

The structural lesson: authentication state shared through a process environment is not configuration, it is an implicit contract between two programs that never talk to each other. When a local stack grows past one client, terminate auth at a boundary instead of inside the model server — a gateway in front of the endpoint validates keys once, in one place, and the server stops having an opinion about callers. That is the same reason a routed endpoint such as 4sapi exists for mixed local-and-hosted model stacks: one credential surface, one place to change a policy, instead of an env var that two processes read differently.


16. What generalizes beyond this one case

Four transferable conclusions, in order of usefulness.

Silent fallback is silent policy. The client did not fail; it applied a default and reported it once, at startup, in a line most people scroll past. Any system that substitutes a generic record for an unknown entity is making a decision on your behalf. Read the startup output before debugging the workload.

Measure the layer, not the symptom. The curl probe with output_tokens: 8192 moved the question from "is the model weak?" to "where is the ceiling?" in one request. Layer-by-layer measurement is cheaper than parameter guessing, and it produces a fact you can act on.

Schema errors cascade by design. Parsers stop at the first violation, so a hand-written record fails serially. Reading the struct names out of the binary converts a guessing loop into a checklist.

A ceiling has two costs. Raising a limit buys output length and spends latency, linearly. The right value is a workflow decision, not a maximum.

One line summarizes the whole case: the client did not know the model, so it applied a fallback record, and 3,208 came from that record — not from the model, not from the server, and not from the 131,072-token context that was there the entire time.

Top comments (0)