DEV Community

MilkyWay008
MilkyWay008

Posted on

Your Windows agent CLI pauses 15 seconds before every command: read the Procmon trace

I keep a list of complaints that sound like vibes and turn out to be real bugs. One of them was "the agent CLI on Windows sits there thinking for 15 seconds before it starts". Not slow streaming...... just dead air between Enter and the first byte of actual work, every single command.

Turns out that one has a trace behind it. There's an open issue on the Codex repo (openai/codex#41351) with the kind of evidence I wish every bug report had. Numbers first: about 15.6 seconds per command in the unelevated Windows sandbox, and roughly 122ms for the same command with the sandbox set to danger-full-access.

Same machine. Same binary. A 128x difference.

That is not your CPU, and it's not your disk.

What the trace actually showed

Procmon, filtered to the process and to CreateFile, sorted by duration, puts almost the whole delay in one row: about 15.42 seconds on a single CreateFile, result OBJECT PATH INVALID, on a path that starts with two backslashes and names the NUL device. The stack runs CreateFileW -> GetDriveTypeW -> ZwCreateFile -> FLTMGR.SYS.

Two things stand out. 1st, the path is a device path that isn't spelled the way it needs to be. 2nd, GetDriveTypeW is sitting in the middle of it, and that function is a known hazard on paths it can't classify. Both of those matter, so here's the background.

The path rules that get missed

Microsoft's naming files, paths and namespaces page is worth the 10 minutes if you ever build paths by hand in Windows code. The short version:

  • NUL is a reserved device name. It works in any directory, which is why > NUL works from anywhere.
  • \\.\NUL uses the Win32 device namespace. That's the explicit way to open a device.
  • \\NUL is not a device path. A path that starts with two backslashes is a UNC name, that is \\server\share. So Windows routes it to the network provider chain (LanmanWorkstation, mrxsmb) and goes looking for a server called NUL. There isn't one.
  • \\?\C:\... is the extended-length prefix. It tells the API layer to skip string parsing and normalization, and it disables device-name translation too.

Then the hazard on top: GetDriveTypeW asks the provider layer what kind of drive a path refers to, and on an unavailable UNC path that call can block for a long time. This isn't theoretical. wxWidgets issue #8859 is the same shape from 2007: wxFSVolume hung for about a minute on an unreachable \\server\drive because its internal FilteredAdd called GetDriveType, and the reporter noted that drive-letter availability gets cached but UNC paths don't, so it hangs on every call. That one got patched in 2022.

So a bogus double-backslash path, plus a function that queries the network provider, equals a stall. The path is the bug. The function is where the time goes.

The actual defect is one over-escaped string

In codex-rs/windows-sandbox-rs/src/acl.rs, the sandbox's allow_null_device() passes this to CreateFileW:

to_wide(r"\\\\.\\NUL").as_ptr(),
Enter fullscreen mode Exit fullscreen mode

That looks like ordinary Windows escaping. It isn't, because r"..." in Rust is a raw string and raw strings don't process escapes. The bytes are literal: four leading backslashes, a dot, then two more backslashes, then NUL. The correct literal for the device namespace has two leading backslashes and one after the dot:

to_wide(r"\\.\NUL").as_ptr(),
Enter fullscreen mode Exit fullscreen mode

I'm not picking on Rust here, the trap exists everywhere verbatim strings do. C# @"...", PowerShell single quotes, raw string literals in most modern languages. You add a layer of escaping to be safe, and the path you ship means something different from the path you meant.

The honest caveat: 15 seconds is not universal

I nearly wrote a wrong paragraph about this, so here it is properly.

On my own Windows 11 box, a malformed device path does not stall. It fails immediately. \\NUL throws out of GetFullPath, CreateFileW returns error 161 (ERROR_BAD_PATHNAME) in under a millisecond, and GetDriveTypeW just says DRIVE_NO_ROOT_DIR. Microseconds, not seconds.

The reporter's machine behaves differently, and their own WPA trace explains why: a third-party filesystem minifilter, 360FsFlt.sys, sits in the path of that failing call and adds about 15.4s of its own. So the malformed path is a real defect either way, but the price of it depends on what is filtering I/O on your machine. Which is also why this is hard to search for. Fast on a clean VM, miserable on a real desk.

What they did confirm with a controlled change is the direction of the fix. Byte-patching that one literal in codex.exe and codex-command-runner.exe, with nothing else changed, took spawn_ready from 23.4s down to 0.6s, then 0.5s and 0.47s on repeats. Same versions, same machine, only the string fixed.

Status while I'm writing this: the issue is still open, no merged upstream fix, and the same literal is still there on main and in the 0.155 alpha I checked. So treat the correction as confirmed by inspection, not shipped.

Diagnosing it on your own machine

This recipe transfers to any "my tool pauses N seconds before doing anything" complaint, which is the main reason I wanted to write it up:

  1. Run Procmon as admin, stop capture (Ctrl+E), clear (Ctrl+X), then start capture just before you reproduce.
  2. Ctrl+L to filter: Process Name is your tool, and Operation is CreateFile.
  3. Options -> Select Columns -> add Duration and Result.
  4. Sort by Duration, descending. One row usually owns the whole delay.
  5. Right-click that row -> Properties -> Stack. If FLTMGR.SYS shows up, you're not looking at app logic anymore, you're looking at a filesystem filter driver.
  6. To name the filter, capture with WPR and read Mini-Filter Delays in WPA, matching the PID and TID from Procmon.
  7. If the stack went through the network provider, correlate the registry and network activity: filter Process Name is System and Path contains LanmanWorkstation.

If your tool stalls and the stack is pure application code, you have a different bug and this writeup won't help you.

A probe you can run in 30 seconds

This demonstrates the path semantics, not the 15 seconds. Add the warm-up call, cold-start noise is real:

Add-Type -Namespace W -Name K -MemberDefinition @'
[DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
public static extern uint GetDriveTypeW(string p);
[DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
public static extern IntPtr CreateFileW(string p, uint a, uint s, IntPtr sa, uint d, uint f, IntPtr t);
'@

foreach ($p in 'NUL', '\\.\NUL', '\\.\\NUL', '\\NUL') {
  [void][W.K]::CreateFileW($p, 0x60000, 3, [IntPtr]::Zero, 3, 0, [IntPtr]::Zero)  # warm up
  $sw = [Diagnostics.Stopwatch]::StartNew()
  $h  = [W.K]::CreateFileW($p, 0x60000, 3, [IntPtr]::Zero, 3, 0, [IntPtr]::Zero)
  $e  = [Runtime.InteropServices.Marshal]::GetLastWin32Error()
  $sw.Stop()
  "{0,-10} {1,7:N2} ms  GetDriveType={2}  err={3}" -f $p, $sw.Elapsed.TotalMilliseconds, [W.K]::GetDriveTypeW($p), $e
}
Enter fullscreen mode Exit fullscreen mode

On my box that prints err 161 for the over-escaped form and for \\NUL, and err 5 for the two device-namespace forms. Err 5 is not a path failure, it means the path resolved and my probe just didn't ask for the right permissions.

What I'd take away from it

  • If you build Windows paths by hand, count the backslashes. \\.\ for devices, \\?\ only for extended-length paths, and never a leading \\ unless you actually mean a server.
  • Don't hand constructed paths to GetDriveTypeW. It asks the provider layer, and that can block.
  • Stuck on Codex specifically? sandbox = "danger-full-access" under [windows] is the measured escape hatch, about 122ms. It weakens the sandbox, so that's a workaround, not a fix.
  • Chasing a stall you can't explain? Trace before you theorize. One Procmon row sorted by duration beats an hour of guessing.

I could be wrong about the exact cost on your machine, since all I have is the reporter's trace and my own clean box to compare against. But the path spelling is checkable, and it is wrong upstream, so that part I'm fairly confident about. The trace and the before/after numbers came from the people in that thread, not from me. I just read it and went, huh, that's the trap I've tripped over before.

Docs I leaned on: Naming Files, Paths, and Namespaces, GetDriveTypeW, the Codex issue, and wxWidgets #8859.

Top comments (0)