Originally published on hexisteme notes.
The warning named my documents: Codex had skipped loading 13 skills because their SKILL.md files were invalid. The detail beneath the heading named a different failure: failed to read file: Too many open files (os error 24).
That distinction determined the repair. A file that could not be opened had not necessarily failed a Markdown or schema check. Editing its contents would not answer why the process could not read it.
This is a postmortem of my September incident, based on the remediation record dated 2026-09-21. The observations below belong to that run. They are not a claim about every Codex release or every warning carrying the word “invalid.”
Start with the nested error
The useful signal was EMFILE, the process-level “too many open files” error. “File” in this context includes descriptors consumed by communication channels. My setup had stdio MCP servers attached to a long-lived Desktop app-server, so ordinary skill documents were sharing a resource budget with pipes.
The diagnostic record contained several different measurements:
| Surface measured | Observation before remediation |
|---|---|
| Shell soft NOFILE limit | 256 |
| Shell hard NOFILE limit | unlimited |
| Kernel-reported maximum number of files per process | 92,160 |
| Desktop app-server numeric descriptors | 274 |
| Desktop app-server PIPE entries | 201 |
| Desktop app-server direct children | 67 |
| Globally enabled MCP servers | 23 |
These are separate surfaces. In particular, the shell's soft limit was not a measurement of the already-running Desktop process's effective limit. Seeing 274 descriptors in that process does not establish that it was running under the shell's 256 limit. I would need the target process's own limit to make that claim.
What the snapshot did show was a substantial pipe footprint alongside the read failures. The process inventory also showed MCP groups corresponding to several sessions. The resource hypothesis therefore had evidence beyond the top-level warning, even though the snapshot alone did not identify the exact descriptor allocation that failed.
My inventory was also broad: 61 skills under the agent tree, 16 under the Codex skill directory, and 112 in plugin caches. Those counts describe the configured surface, not the number of simultaneously open files. I did not turn them into a concurrency measurement.
Two pressures met at the same boundary
The incident record connected the symptom to parallel skill loading and the pipes associated with session-specific MCP children. It referenced the upstream discussions in issue #36755, issue #26984, and issue #37971. Those are investigation pointers from the record; their current status is not evidence supplied by this post.
My working explanation was that loading skills needed transient descriptors while the long-lived process already held a large communication footprint. A document happened to be the next resource consumer denied an open. Its filename made the failure look local to the document, while the nested error implicated the process.
I treated that explanation as sufficient to test a resource intervention. I did not have a controlled experiment separating the contribution of loader concurrency from the contribution of retained MCP pipes. The successful intervention below supports the resource diagnosis, but it cannot assign each mechanism a share of the failure.
Change both the budget and the default demand
I changed the CLI launch path to request a soft NOFILE limit of up to 65,536 before starting Codex. The wrapper respects a lower hard limit and preserves its existing behavior if the operating system refuses the change. The domain launcher received the same target.
There is a limitation in my own evidence here. The verification note records a wrapper probe issuing ulimit -Sn 4096 from a parent whose soft limit was 256. That supports the narrower claim that the launch path could attempt a higher limit. It does not prove that the final 65,536 target was effective in every launched process. Configuration intent and an inherited runtime limit are different facts, and my record does not close that gap.
The other change reduced routine demand. I cut the global MCP default from 23 servers to 8, disabled the Comfy and video-vision plugins by default, and retained domain profiles for work that needed a larger tool set. The full profile retained the previous broader configuration.
That gave me an operational choice: a smaller baseline for ordinary work, with broader dependencies selected when required. A high ceiling alone would have left the original pipe demand in place. A smaller baseline alone would not have addressed the CLI's low inherited soft limit.
The GUI path had its own boundary. My attempt to change the launchd maxfiles limit from the user session was rejected with Operation not permitted. I did not record that as a successful Desktop limit increase. The changes actually applied were the reduced default demand and the CLI-side launch behavior.
Measure the process after the change
After hot reload, the recorded Desktop app-server snapshot had 116 numeric descriptors, 78 PIPE entries, and 26 direct children. The before-and-after snapshots show that the retained footprint fell. They do not establish that an upstream lifetime-management defect was permanently fixed.
The validation record was broader than the snapshot. Shell syntax checks passed for the wrapper and launcher. All 11 profile configurations passed their parsing and MCP-list checks. The strict doctor run reported 23 checks OK.
I also ran fresh ephemeral executions with both the base profile and the full profile. Each returned exit 0, produced the requested response marker, and emitted no FD or skill warnings. The check after those runs found no new MCP orphans.
Those fresh runs are the strongest recovery evidence in the record: the system performed the operation that had been failing. The full-profile run matters because reducing the default server set was not the only path exercised. Still, these were bounded smoke checks, not a long-duration test of session churn. “Recovered in the checked runs” is supported. “Cannot recur” is not.
Keep a way to reject the diagnosis
The remediation record includes a specific falsifier, evaluated on 2026-09-21: if a fresh process with the base profile and a verified NOFILE limit of 65,536 repeatedly fails on the same SKILL.md files after os error 24 disappears, reclassify those files for a separate syntax or schema defect.
The word “verified” matters because of the wrapper-probe gap. I would confirm the effective limit before treating that retry as a test of the falsifier. Otherwise, a failed attempt to apply the mitigation could be mistaken for evidence against the diagnosis.
For recurrence, my runbook starts with the Desktop app-server's pipe count and direct-child count. Growth across sessions under the smaller baseline would reopen the lifetime-management question. Restarting the application is a recovery action in that situation, not proof that the cause has been removed.
A fresh session also matters after a skipped-skill warning. The record's recovery procedure opens a new session rather than assuming the failed load has been repaired inside the old one. Saving active work and fully restarting Desktop is the stronger reset when the affected process persists.
The diagnostic process had a separate failure
During this investigation, raw configuration diffs and adjacent configuration lines exposed credential values in tool output. The remediation record explicitly leaves credential rotation as an unresolved follow-up. I am not reproducing those values or treating the descriptor fix as a security repair.
The operational lesson is concrete: configuration diagnosis should print server names, enabled flags, counts, and whether credentials are present. Narrow line ranges are not a sufficient filter when secrets can occupy neighboring lines. That mistake belongs in this postmortem because it happened while collecting the evidence, not because it explains EMFILE.
The resource repair began when I stopped taking “invalid” as the diagnosis and followed the nested read error. The acceptance check was equally specific: fresh executions completed without the original warning, and the observed pipe footprint fell. Keeping those claims narrower than “all limits fixed” leaves the next investigation somewhere honest to start.
This post was prepared with AI assistance from my incident and remediation record.
License: CC BY 4.0.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (4)
The concurrency spike on loader startup is where this bit us. An unthrottled directory walk reading dozens of skill manifests in parallel hits peak file descriptors within the first two hundred milliseconds, right when stdio transports are still establishing their pipe pairs. Putting an eight-worker limit on the file reads kept peak descriptor demand flat without touching the system limits.
That eight-worker cap on the manifest walk is the sharper fix — it targets the exact 200 ms window where the loader and the stdio pipes collide, instead of raising the ceiling everywhere. I only measured the symptom; you isolated the race. Thanks for naming the concurrency spike and the worker limit that flattens it.
the part that stands out is you separated the shell's soft limit from the actual running process limit instead of assuming they matched. most people would have seen 256 and stopped there. curious if the 8 vs 23 default MCP servers change alone would have fixed it without the NOFILE bump, or if you needed both together to actually confirm it
That distinction bit me because the shell reported 256 but the traced process showed 1024 — assuming they matched would have sent me down the wrong path. The 8-server default likely would have kept things under the soft limit, but I needed the NOFILE bump to prove the hard ceiling was the real constraint, not the configured count. Thanks for catching that the validation required both the config change and the limit raise together.