I Built lls, a Go CLI to List 33.12 Million Files

Sometimes a problem looks simple at first.

In my case, I needed a complete file list from a huge directory on storage mounted over NFS from an application server. At first, this sounded like something existing tools should be able to handle. But once the number of files became extremely large, that assumption stopped being true.

I eventually built a Go CLI called lls to solve this problem.

This was not a toy project. I built lls to solve a real production problem, and in the end it was able to list 33.12 million files from a single directory on NFS.

Repository:

https://github.com/catatsuy/lls

In this article, I will explain what failed, why I decided to use the Linux getdents64 system call directly, how the implementation works, and how lls finally solved the problem.

The problem

The directory I had to deal with was on storage mounted over NFS, and it contained an extremely large number of files.

If a directory is small, ls and find are usually enough. But once the number of files becomes too large, even getting a complete file list becomes difficult. And when the directory is on NFS instead of local storage, the situation can become even worse.

What I needed was simple in theory: get the full list of files and finish successfully. In practice, that turned out to be the hard part.

ls -U1 and find could not finish

The first thing I tried was ls -U1.

I disabled sorting because sorting is one of the well-known reasons ls becomes painful on huge directories. But even with ls -U1, it still could not finish. The number varied from run to run, but at best it stopped after outputting about 6 million files.

I did not fully investigate why it stopped, but I suspected the storage server might have stopped responding.

Next, I tried find.

I thought find might handle more entries than ls, but it also failed. The result also varied, but at best it output about 12 million lines before it stopped responding.

At that point, I was almost ready to give up. I started thinking I might have to output part of the file list, delete files in multiple rounds, and somehow work around the problem manually.

But I wanted a real solution.

Why I built lls

Around that time, I found an article describing how someone listed a directory containing 8 million files by calling getdents directly with a large buffer. That was the key idea I needed. The article showed the C approach, but not a ready-to-use implementation, so I decided to build my own tool in Go.

http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html

In Go, I could call Linux-specific system calls through the syscall package. That meant I could stay in Go, avoid cgo, and still work directly with the kernel interface I needed.

That was how lls started.

The point of lls was not to replace ls in general. It was a narrow tool for one difficult job: keep reading directory entries from a huge directory until the end.

System calls and getdents64

On Linux, userland programs ask the kernel to do work through system calls. Directory reading is no exception.

For this problem, the important system call was getdents64. The older getdents exists, but getdents64 was added because the original interface did not handle large filesystems and large file offsets well. In Go, the function exposed as syscall.Getdents uses getdents64, which was exactly what I needed here.

The returned data is not a high-level file list. It is raw directory-entry data packed into a byte buffer.

Conceptually, the data corresponds to a structure with fields such as inode number, offset, record length, type, and a null-terminated file name.

That detail matters, because if you use getdents64 directly, you have to parse the buffer yourself.

My first idea: one large buffer

My first idea was simple:

1. allocate a buffer based on the directory size
2. call getdents64
3. print each file name to standard output

The directory size here was the same value you can see with ls -dl. The idea was that this value should be large enough to hold the full result. If the buffer was smaller than what was really needed, the output would be incomplete.

lls also had a -buf-size option so I could adjust the size manually, and a -debug option to show how much of the buffer was actually used.

However, on the real directory, this did not work as expected.

The directory size reported by ls -dl was over 2 GB, and running lls with that default buffer size produced EINVAL. After trying different values, I found that 2147483647 worked but 2147483648 did not. Later, I concluded that this was because the size had to fit in an int, which also explains why the call failed beyond that point.

Even after increasing the buffer size as much as possible, that approach still was not the real solution. The important point was not “make one call with a bigger buffer.” The real solution was to change the design.

The real fix: call getdents64 repeatedly

The real fix was to stop thinking in terms of one huge call.

getdents64 can be called repeatedly. If you keep calling it until it returns 0, you can continue reading the remaining directory entries.

This became the key change in lls.

Instead of relying on a single enormous buffer, lls now uses a reasonable buffer and keeps calling syscall.Getdents until the directory is fully consumed. That change made it possible to list all 33.12 million files.

That was the point where lls became a practical tool for extremely large directories rather than a one-shot experiment.

The core implementation

The implementation in lls is built around syscall.Dirent, which Go defines on Linux with fields like inode number, offset, record length, type, and a fixed-size name field.

The core loop is straightforward:

• allocate a buffer
• call syscall.Getdents(int(f.Fd()), buf)
• if the return value is 0, stop
• otherwise parse the returned bytes entry by entry

The most important part is how the parsing works.

The returned buffer contains multiple directory entries. Each entry has a variable size, so the code cannot move by a fixed structure size. Instead, it casts the current position in the buffer to *syscall.Dirent, reads Reclen, and moves forward by that many bytes.

That is how it walks through the buffer correctly.

The code also checks Ino. If the inode number is 0, the entry is skipped, because that means the file no longer exists.

For file names, the implementation uses the Name field from syscall.Dirent. In Go this is [256]int8, so the code first treats it as bytes and then converts the bytes before the terminating null into a string.

In other words, the implementation stays intentionally close to the kernel interface:

• call getdents64
• interpret the returned bytes as directory entries
• move forward using Reclen
• extract the file name
• repeat until getdents64 returns 0

Why this worked

One especially useful detail is that libc's readdir implementation often uses a fixed internal buffer. In one of the articles I read, the example used a 2048-byte buffer internally. If your directory is huge, that means a large number of system calls just to read through it.

You cannot easily change that buffer size from outside, which is why directly calling getdents64 yourself can make sense in an extreme case like this.

That does not mean low-level code is always better. It only means this particular problem was narrow enough, and extreme enough, that the lower-level interface matched the problem better than a general-purpose tool.

The result

In the end, the comparison looked like this:

• ls -U1: about 6 million files
• find: about 12 million lines
• lls: 33.12 million files listed successfully

That was the result that mattered.

This was not just an experiment in system programming. It solved a real production problem on NFS. I needed the full file list, and lls made that possible.

Conclusion

I built lls because standard tools could not finish the job on a huge NFS-mounted directory.

The important ideas were:

• use the Linux getdents64 interface through Go's syscall.Getdents
• parse the returned directory entries directly
• advance through the buffer using Reclen
• keep calling the system call until it returns 0

That approach finally made it possible to list 33.12 million files.

If you want to look at the code, here is the repository again:

https://github.com/catatsuy/lls

Useful references:

https://man7.org/linux/man-pages/man2/getdents.2.html

https://pkg.go.dev/syscall