I built a native Mac storage analyzer that can search 2 million files in 2.35 ms

Shivam Chaturvedi — Tue, 10 Nov 2026 12:40:35 +0000

Why I used a flat array to handle millions of files in a macOS storage analyzer

When you're building a disk analyzer, scanning files is only half the problem.

The other half is figuring out how to represent millions of filesystem entries in memory without making every subsequent operation expensive.

I'm building a native macOS disk analyzer called Diskfern. During development, one engineering problem kept coming up:

How should I represent millions of files and folders so that searching, filtering, sorting, and visualization remain responsive?

I initially explored the conventional hierarchical approach: representing directories as nested objects containing their children.

It looks natural because a filesystem is already a tree.

But there is a difference between representing a filesystem and efficiently processing millions of filesystem entries.

Here's the approach I used and why it made sense for this particular workload.

1. The obvious approach: nested objects

A filesystem naturally looks something like this:

Macintosh HD
├── Applications
│   ├── Xcode.app
│   └── Safari.app
├── Users
│   └── Developer
│       ├── Downloads
│       ├── Projects
│       └── Library
└── System

A straightforward representation might look like:

final class FileNode {
    let name: String
    let size: Int64
    var children: [FileNode]

    init(name: String, size: Int64) {
        self.name = name
        self.size = size
        self.children = []
    }
}

For smaller directory structures, this is perfectly reasonable.

However, when dealing with millions of entries, I wanted to avoid creating and maintaining a huge number of independently allocated objects and nested collections.

There are also practical considerations:

Object allocation overhead
Pointer chasing during traversal
Memory fragmentation
Repeated hierarchy traversal
Additional work when building search and visualization data

The hierarchical model was convenient, but I wanted a representation optimized for processing rather than just mirroring the filesystem.

2. The alternative: a flat array

Instead of storing every directory as an object containing other objects, I used a flat collection of file records.

Conceptually:

struct FileRecord {
    let name: String
    let path: String
    let size: Int64
    let parentIndex: Int32
    let isDirectory: Bool
    let modifiedAt: Date
}

And the collection:

var records: [FileRecord] = []

Every file and directory becomes an individual record.

The important part is parentIndex.

Instead of storing a reference to a parent object, a record can refer to its parent using an integer index.

For example:

Index    Name           Parent
--------------------------------
0        Macintosh HD   -1
1        Applications    0
2        Xcode.app       1
3        Safari.app      1
4        Users           0
5        Developer       4
6        Projects        5
7        Downloads       5

The hierarchy is represented through relationships between records rather than nested objects.

This is a simplified illustration of the data model, not a complete production implementation.

3. Why this representation is useful

A flat array changes how you work with the data.

Contiguous storage

Arrays provide sequential storage, which can improve cache locality when processing records in order.

Integer-based relationships

An integer index is cheaper to work with than navigating through a chain of object references.

Simpler bulk operations

Filtering, sorting, aggregating and processing records can happen over a single collection.

Less structural overhead

You don't need a separately allocated node object for every entry just to maintain the hierarchy.

One important distinction: a flat array doesn't automatically eliminate memory overhead or guarantee better performance. Swift arrays, strings and other stored values still have their own costs.

The actual benefit depends on the representation, allocation patterns and operations you're performing.

4. But how do you get the hierarchy back?

This is where things get interesting.

A flat collection doesn't mean you lose the ability to represent directories.

You can reconstruct relationships using parent indices.

For example, if you want to inspect the children of a particular directory, you can find records whose parentIndex matches that directory's index.

For production workloads, repeatedly scanning the entire collection for children would be wasteful.

You can maintain an additional adjacency index:

Parent index → Child indices

0 → [1, 4]
1 → [2, 3]
4 → [5]
5 → [6, 7]

Now you can navigate the hierarchy without maintaining deeply nested object graphs.

The same underlying records can support different representations depending on what the UI needs.

5. Searching millions of records is another problem

Representing records efficiently doesn't automatically make searching fast.

I also needed a search path that could handle queries such as:

PDFs modified last month
Videos larger than 1 GB
Files belonging to a particular directory
Old files that haven't been accessed recently

Diskfern's search engine uses optimized matching and Top-K result selection rather than treating every query as an opportunity to construct a new hierarchy.

In my testing on an M4 MacBook Air with 16 GB RAM:

Operation	Measured time
Scan approximately 2.9 million files	~16 seconds
Search 2 million indexed records	~2.35 ms
Restore 500,000 saved scan items	~18.6 ms

These are results from my own testing environment, not universal performance guarantees.

6. The bigger lesson

The main thing I learned is that the data structure that most closely resembles your domain isn't necessarily the best representation for every operation.

A filesystem is hierarchical, but searching, filtering, sorting and aggregating millions of entries are largely collection-processing problems.

Separating the underlying representation from the way the data is presented gave me more flexibility.

A flat collection can be useful when you need:

Large-scale sequential processing
Efficient filtering and sorting
Compact relationships through integer indices
Multiple views over the same dataset
Predictable data access patterns

It isn't a universal replacement for trees. Some operations are naturally hierarchical, and nested structures remain useful when they simplify the problem.

The important thing is choosing a representation based on the workload rather than just the shape of the domain.

I'd like to hear from other engineers

If you've worked with large datasets, filesystem indexes, or in-memory representations:

Would you use a flat array, an object graph, or an adjacency-list approach for this workload?
How would you optimize parent-child lookups when the dataset contains several million records?
Would you consider memory-mapped storage or a database once the dataset grows beyond available RAM?
What would you benchmark before choosing between these approaches?

I'm particularly interested in approaches that work well in native macOS applications.

I'm building this as part of Diskfern, my macOS storage analyzer: https://www.diskfern.com/

Thanks for reading. I'd genuinely appreciate technical criticism and alternative approaches.

I'm curious how other developers approach this problem.

When you're working with millions of filesystem entries, would you choose a nested object graph, a flat array with parent references, or a different structure entirely?

I'd especially like to hear about the trade-offs you've encountered in real projects.

Disclosure: I used AI assistance to edit and organize this article. The implementation experience and benchmark results are from my own project.

DEV Community: Shivam Chaturvedi