Debugging a Filesystem Module: When Reference Counting Goes Wrong

As I've been working my way through Linux kernel development,I decided it was time to tackle something that seemed simple on the surface: write a minimal filesystem module. How hard could it be to mount a filesystem that contains a single file you can cat? Turns out, pretty educational.

The Goal

I wanted to build the smallest possible virtual filesystem. No disk backing. No persistence, just cat a static file that is generated by the module.The whole thing should live in RAM, expose one file called "hello" that returns some text.

Seems like the next natural step. I mean, how different could it be?

The First Attempt

I started by doing what seemed obvious: create a superblock in fill_super, manually allocate inodes for the root directory and my hello file, create dentries for them, link everything together. Standard VFS stuff. The code compiled. The module loaded. I could mount it. I could even cat the file and see my message.

Then I tried to unmount.

[  337.050239] gs_fs: superblock kill called
[  337.050258] ------------[ cut here ]------------
[  337.051811] BUG: Dentry still in use (1) [unmount of gs_fs gs_fs]
[  337.053385] WARNING: CPU: 0 PID: 72 at fs/dcache.c:1590 umount_check+0x56/0x70

The kernel was not happy. "Dentry still in use" means I left references dangling somewhere. The VFS couldn't clean up properly because something was still holding onto my hello file's dentry.

Down the Rabbit Hole

The error message told me exactly what was wrong but not why. I had to understand the lifecycle of dentries and inodes and their reference counting, and how the VFS expects you to clean up during unmount.

First theory: maybe I needed to implement evict_inode. So I added a proper super_operations struct with an evict callback that calls truncate_inode_pages_final() and clear_inode(). That's the standard pattern for cleaning up inodes (so it seems to me, correct me if I'm wrong PLEASE!).

Nope.

Second theory: maybe it's how I was creating the dentries. I was using d_alloc_name() to manually create the dentry for my hello file during mount. That gives you a dentry with a reference count, and there's no automatic mechanism to drop it. The VFS doesn't know about dentries you create manually like that (again, PLEASE set me straight if that's not the case).

But here's the thing, I wasn't just randomly guessing. I started looking at how other simple filesystems do it. And that's when I found simple_fill_super(). Probably should start reading more of the kernel docs, I guess?

The Kernel's Helper Functions

Turns out the kernel has a bunch of helper functions specifically for pseudo-filesystems like mine. simple_fill_super() takes an array of file descriptors and sets up all the dentries, inodes, and reference counting for you automatically. It handles the lifecycle properly.

So I refactored to use it:

static int gs_fs_fill_super(struct super_block *sb, struct fs_context *fc)
{
    static const struct tree_descr files[] = {
        { HELLO_FILENAME, &gs_hello_fops, 0444 },
        { "" }  // Sentinel
    };

    sb->s_op = &gs_fs_super_ops;
    return simple_fill_super(sb, GS_FS_MAGIC, files);
}

Mounted it. Cat'd the file. Worked great. Tried to unmount.

Nope.

The Real Problem

At this point I was getting frustrated. I had the right helpers. I had proper cleanup. What was I missing?

Then I looked more carefully at my kill_sb function:

static void gs_fs_kill_sb(struct super_block *sb)
{
    pr_info("gs_fs: superblock kill called\n");
    kill_anon_super(sb);  // This was the problem
}

I was using kill_anon_super() because I saw it in some example somewhere and it seemed reasonable. Anonymous superblock, right?

When you use get_tree_nodev() with simple_fill_super(), you need to use kill_litter_super() instead. kill_litter_super() knows how to properly clean up structures created by simple_fill_super(). It handles all the dentries and inodes that got set up by that helper.

Changed one line:

static void gs_fs_kill_sb(struct super_block *sb)
{
    pr_info("gs_fs: superblock kill called\n");
    kill_litter_super(sb);  // Fixed
}

Perfect!

Why This Matters

This bug taught me more about the VFS than any amount of documentation reading could have (entirely speculation here, as I can't actually read). I had to dig into:

• How dentries cache the filesystem namespace
• How reference counting prevents premature cleanup
• Why the kernel provides helper functions and when to use them
• How different superblock types need different cleanup strategies

The kernel has these subtle API pairings all over the place. Use get_tree_nodev()? Pair it with kill_litter_super(). Use simple_fill_super()? Make sure your super_operations are set up properly. The compiler won't catch these mismatches because they all compile just fine. You only find out at runtime.

A valuable set of lessons taught by getting my hands dirty.

What's Next

Now that I have a working minimal filesystem, the obvious next steps are:

• Implement write support
• Add subdirectories
• Make files appear on-demand via .lookup

Not sure I'll continue on the filesystem path or divert, but we'll see.