Every entry twice, then nothing at all: debugging directory listings on macFUSE's FSKit backend

I build ghfs, a small tool that makes GitHub Issues readable as local files, so a coding agent can read an issue with cat instead of calling an API mid-task. On Linux it sits on libfuse. On macOS it mounts through macFUSE, using its FSKit backend (-o backend=fskit), which runs the file system as a File System Extension instead of a kernel extension.

This post is about one Readdir function on macOS, three bugs that lived in it, and a root-cause conclusion that I wrote down in April and that turned out to be wrong.

Symptom 1: every entry, twice

The first report was simple. Run tree on the mount twice, and the second run lists every entry twice. The first run is correct. Linux never shows it.

I tried the things you try:

• passing sequential offsets to fill instead of 0
• the novncache mount option (ignored by the FSKit backend)
• a stable mtime for directories, which I had been setting to time.Now()
• a unique directory handle from Opendir
• stable inode numbers derived from the path
• going back to the style of cgofuse's memfs example

The stable mtime was a real fix and made the first run correct. None of the rest changed the second run.

So I wrote down a root cause: FSKit accumulates readdir results. Return the same entries twice and FSKit unions them. Then I built a workaround on top of that belief:

1. On macOS, when Readdir is called with a non-zero offset, return nothing. Treat it as the tail of a listing already served.
2. Keep a short-lived cache (500 ms). If the same directory is listed again inside that window, return nothing, so FSKit has nothing to add.

The duplicates went away in the scenario I cared about. One test, tree twice with nothing else before it, still failed, so I skipped it on macOS and opened a follow-up issue.

Symptom 2: nothing at all

Months later, while testing a release, the follow-up turned out to be bigger than tree. Listing the same directory twice within half a second returned an empty directory:

first ls        → test-project
ls after 0.2 s  → (empty)
ls after 0.4 s  → (empty)
ls after 1.9 s  → test-project

For a tool whose readers are coding agents, this is the worst possible failure. An agent that lists the issues directory twice in a row sees no issues, and nothing anywhere reports an error.

Looking at what actually arrives

I logged every FUSE call for the root directory during two back-to-back tree runs:

time	call	what ghfs did
+0 ms	Opendir fh=1	released right away
+5 ms	Opendir fh=2
+65 ms	Readdir fh=2 ofst=0	returned 3 entries (first tree)
+65 ms	Readdir fh=2 ofst=3	returned nothing (continuation)
+91 ms	Readdir fh=2 ofst=0	cache hit, returned nothing (second tree)

Two things stand out. The second tree does reach ghfs, so FSKit is not answering from its own cache. And Opendir is not called again: the second listing reuses the first handle. The macFUSE maintainer has explained why. The macOS VFS does not pass file or directory handles to file systems, so macFUSE emulates them and reuses them, and two readdirs on the same directory share one handle.

Then I turned the 500 ms cache off, and the duplicates came back.

That is the trap. The listing that must get entries (a new tree) and the listing that, under my theory, had to get nothing arrive in exactly the same shape: same handle, offset 0, tens of milliseconds after the previous one. To avoid the empty case the cache window had to be under 6 ms. To avoid the duplicate case it had to be over 56 ms. No value of the timeout satisfies both, and nothing in the handle, the offset or the open/close sequence tells them apart.

Symptom 3, hiding in plain sight

To tell theories apart I wrote a measurement: put N entries in a directory, list it three times, record duplicates and gaps. I expected large directories to show duplication partway through. Instead:

entries created	entries listed	duplicates
64	63	0
100	63	0
200	63	0
500	63	0

Every listing stopped at 63. That was workaround step 1. When the reply buffer fills, the caller comes back with a non-zero offset to get the rest, and I was answering "nothing". Reading readdir with a non-zero offset is a normal part of the FUSE contract, and returning nothing there is simply wrong. On a real mount, a directory of 344 closed issues listed 255.

Two of my three bugs were the workaround.

Doing readdir properly

So I removed both workarounds. Readdir now builds the full listing, starts from the offset it was given, and passes sequential offsets to fill so the caller can resume:

total := int64(len(all))
start := ofst
if start < 0 {
    start = 0
}
if start > total {
    start = total
}
for i := start; i < total; i++ {
    e := all[i]
    fillOfst := int64(0)
    if runtime.GOOS == "darwin" {
        fillOfst = i + 1
    }
    if !fill(e.name, e.stat, fillOfst) {
        break
    }
}

With macFUSE 5.3.3, large directories now listed completely: 500 entries, three times, no gaps and no duplicates. The offsets ghfs handed out came back unchanged, so the resume path works. But overlapping listings still duplicated. A walk over three issue files returned 27.

The experiment I should have run in April

macFUSE 5.4.0 had just come out, and its release notes say the FSKit backend now improves directory enumeration by "validating cookies and verifiers". I swapped macFUSE on the same machine and ran the same ghfs binary against it:

test	macFUSE 5.3.3	macFUSE 5.4.0
500 entries, listed 3×	500, no duplicates	500, no duplicates
scenario suite	fails (3 files → 27)	passes
tree twice	outputs differ	identical
the test skipped since April	(skipped)	passes, 3 runs in a row
same directory at 0 ms and 200 ms	—	20 of 20 entries every time

The ghfs binary was the same. Only macFUSE changed, and the duplication disappeared. So the duplication was never ghfs, and it was not "FSKit accumulates results" either: FSKit was the same on both runs. It was the translation between FUSE readdir and FSKit's enumeration inside macFUSE, which 5.4.0 changed. The maintainer has also described elsewhere how the FSKit API does not say how many entries it wants, so the backend can end up re-requesting entries it has already returned.

My April conclusion had a plausible mechanism, and it survived six failed fixes because each fix was tested against it rather than against an alternative. The experiment that could falsify it was cheap: hold my code still and change the layer below. I didn't run it until September.

What this did to the requirements

The fix only holds on macFUSE 5.4.0 or later, so that became a requirement. That version also fixes a crash where an empty read reply took down the File System Extension, and ghfs could send one.

While I was in there, I checked the macOS requirement too. macFUSE ships two FSKit extensions. One is for "local" volumes, meaning physically attached storage, and runs on macOS 15.4 and later, but mounting that way needs an option macFUSE itself calls experimental. The other, for everything else, requires macOS 26. ghfs uses the second one. My docs said "macOS 15.4 or later", and nobody had ever checked it: CI runners cannot enable a File System Extension, because that needs a person to approve it in System Settings.

So v1.1.0 requires macOS 26 or later and macFUSE 5.4.0 or later. A narrower promise that has been checked is worth more than a wider one that hasn't.

One more FSKit lesson: a leftover mount you cannot touch

The same release fixed one more macOS behaviour, which is worth knowing if you build on macFUSE. If the daemon is killed abnormally, macFUSE's file system module keeps holding the volume with nothing behind it. Anything that touches that path, ls included, enters an uninterruptible wait and cannot be killed. mount, df and lsof walk the mount table and hang the same way. umount does not return, and umount -f removes the entry but leaves the module, so mounting at the same path fails.

Two things do work. getfsstat with MNT_NOWAIT returns in about 10 µs and still lists the stale mount, so detection is safe. Sending SIGTERM to the file system module clears everything, with no reboot.

What I could not find is a way to tell which module process holds which volume. Quitting the module affects every macFUSE volume on the machine, so ghfs now detects the leftover mount, stops before trying to mount, and prints the command (pkill -f io.macfuse.app.fsmodule.macfuse) for the user to run. The old code had assumed FSKit unmounts automatically when the process dies. It doesn't.

Takeaways

A workaround built on a wrong root cause spreads. Mine produced two new bugs, and one of them was silent.

When a symptom might live in the layer below you, hold your code still and change that layer. It is the cheapest experiment you have.

Write the root cause down as a hypothesis until something has tried to break it.

On FSKit, readdir at a non-zero offset is normal. Resume from it.

---

ghfs is what all this was for. It prefetches GitHub Issues into a read-only mount so a local coding agent can read them as ordinary files. It needs macOS 26 or later on Apple Silicon with macFUSE 5.4.0 or later (install macFUSE and enable its File System Extension in System Settings before installing ghfs), or Linux on x86_64 or ARM64 with libfuse3, including WSL 2. It is paid and not open source, and the free tier covers one repository: https://ghfs.dev

Comments

[Nathan]

@ghfs_dev Your reply reached my inbox but it is not visible on the post right now (the permalink dev.to sent me, /ghfs_dev/comment/3fhjp, 404s). Flagging it in case your comment got eaten on the way out.

The part I wanted to answer is the discipline in it: you wrote the prediction down before you looked. If April's theory is right, this directory lists three times, duplicates partway through. Then the clean stop at 63 has nowhere to hide. The check was built so it could kill the theory, not confirm it.

That is the step I usually skip. I write the check after I already believe the fix, so it is shaped to pass. The one I trust now is the one I run against deliberately broken code: if the suite stays green, it never touched the thing.

Same disease as "two of my three bugs were the workaround". The workaround encodes the wrong theory, so it hides the real bug and it survives every test you write to confirm the theory.

[ghfs.dev]

Thanks for flagging it. Nothing got eaten on dev.to's side: I deleted it myself. When I looked at the post again after replying, your comment appeared to have been deleted, so I took my reply down too.

To be fair to the April version of me, I didn't predict the 63. I expected duplication partway through, and the measurement caught a bug I wasn't looking for. What it did right was record every entry instead of checking for the one outcome I expected. The check that actually killed the April theory came later: hold ghfs still and swap macFUSE. Your broken-code test is the same move from the other side. Keep one side fixed, change the other, and see whether the result follows.

[Nathan]

Fair correction, and I'll take it. The prediction was not the valuable part. You didn't predict the 63, and it still showed up because the table recorded every entry instead of the one result you were testing for.

That is the part I usually get wrong. My check would have been 'does the path resolve to one key', which the read-path test answered yes to for months. Nothing printed the keys the write tools actually built, so the duplicate sat there untested. A check that only collects the outcome you expect stays green while the bug walks past it.

The swap-one-side version is cleaner than what I do. Hold everything still, change one thing, see if the result follows. Good to know nothing got eaten on your side; my comment went invisible for a stretch too and is back now.