The Filesystem Your Multi-GPU Rig Actually Needs

You install four PCIe Gen5 NVMe drives in a multi-GPU server, bring up eight H100s, launch torchrun, and notice an irritating pattern: GPU utilization fluctuates between 40% and 95%.

A quick look at nvidia-smi suggests that the GPUs are waiting for data. Profiling shows that CPU time is being spent not only on preprocessing, but also on filesystem activity: metadata operations, lock contention, and page-cache writeback during checkpointing.

In a modern deep-learning system, the filesystem is not merely a passive storage layer. It is part of the data pipeline, and its concurrency characteristics can affect end-to-end training throughput.

This article looks at two common local-storage workloads:

1. Parallel data ingestion: dozens of PyTorch DataLoader workers issue concurrent reads and metadata requests.
2. Large checkpoint writes: training processes serialize model weights, optimizer states, and tensor data into files that can range from 10 GB to more than 200 GB per rank.

For these workloads, XFS is often a strong choice for local NVMe scratch storage. Its allocation-group architecture, extent-based allocation, and delayed allocation can make it scale well under concurrent access. That does not mean XFS wins every workload or that filesystem choice alone fixes an input pipeline; the data layout, access pattern, kernel, mount options, and hardware all matter.

At a glance

Workload or concern	Why it matters	XFS characteristics
Many concurrent readers	Workers compete for metadata and I/O resources	Allocation groups allow metadata work to be distributed
Large sequential writes	Checkpointing can interrupt training	Extent-based allocation and delayed allocation help produce large contiguous allocations
Millions of tiny files	Metadata operations can dominate useful I/O	XFS is capable, but a sharded dataset is usually a better architectural solution
Copy-on-write workloads	Shared extents can add write and metadata overhead when modified	Ordinary XFS writes are in-place; reflink-shared extents use CoW
NVMe throughput	Queueing and I/O size affect achievable bandwidth	XFS can scale well, but device and workload tuning still matter

1. The concurrency problem

PyTorch's torch.utils.data.DataLoader can run multiple worker processes using num_workers. On a multi-GPU node, the total number of workers can become substantial. For example, eight training processes with 16 workers each can create 128 independent workers.

Those workers may concurrently:

• Open and close files.
• Read file and directory metadata.
• Resolve paths and look up inodes.
• Issue data reads.
• Update access times, depending on mount options.

The resulting workload is not simply a stream of large reads. It can also be a high-rate stream of small metadata operations.

A simplified view of the I/O path

PyTorch DataLoader workers
  Worker 1   Worker 2   Worker 3   ...   Worker N
      \         |         |                /
       \        |         |               /
        +-------+---------+--------------+
                        |
                        v
              Linux Virtual Filesystem
                        |
             +----------+----------+
             |                     |
             v                     v
           ext4                   XFS
     Block groups and       Allocation groups
     journal coordination   with per-AG metadata

This is a conceptual diagram, not a claim that every operation is serialized by one global lock in ext4 or that XFS eliminates locking. Both filesystems use sophisticated concurrency mechanisms. The important distinction is that XFS can distribute many allocation and metadata operations across allocation groups.

2. Why ext4 can become a bottleneck

ext4 is a mature filesystem. It performs very well across a wide range of workloads. Under highly concurrent metadata workloads, however, contention can arise around shared filesystem structures and journal activity.

Two areas are worth understanding:

• Journal activity: metadata updates are coordinated through JBD2 transactions. A workload that creates, renames, or updates large numbers of files can generate substantial journal traffic.
• Allocation and metadata coordination: concurrent inode and block allocation can contend on shared structures, depending on the access pattern and filesystem layout.

The practical point is not that ext4 has a single global mutex. It does not. Rather, the cost of coordinating metadata operations can become visible when many processes issue small, concurrent filesystem requests.

3. How XFS uses allocation groups

XFS was designed for large filesystems and high-throughput workloads. One of its central design features is the allocation group (AG).

An XFS filesystem is divided into allocation groups. Each AG maintains its own allocation metadata, including structures used to track free space and inodes. These structures are commonly implemented using B+ trees.

An allocation group maintains, among other things:

• Free-space information, indexed by block location and extent size.
• Inode allocation information.
• AG-local metadata and coordination structures.

When multiple workers perform allocation or metadata operations, XFS can distribute work across different AGs rather than forcing all such activity through one shared allocation structure.

This does not make XFS lock-free. It means its architecture provides multiple allocation domains, which can reduce contention for workloads that exercise them effectively.

Practical implication: XFS is worth benchmarking when a node has many concurrent workers accessing a local NVMe filesystem, particularly when metadata activity is a measurable part of the bottleneck.

4. Checkpointing

Saving a modern deep-learning checkpoint can involve writing tens or hundreds of gigabytes of tensor data. Examples include:

• PyTorch state dictionaries.
• Optimizer state.
• Distributed training or ZeRO partitions.
• BF16 or FP8 tensor dumps.

The goal is to complete checkpoint writes quickly and predictably, minimizing the time training is interrupted or competing with storage activity.

Extent-based allocation

XFS uses extents to describe ranges of filesystem blocks. Instead of representing a file as a long list of individual block pointers, an extent can describe a contiguous range.

For large sequential writes, this can keep file metadata compact and help the filesystem allocate large runs of blocks. The actual extent layout still depends on free space, concurrent activity, write patterns, and filesystem state; contiguity is not guaranteed.

Delayed allocation

With buffered I/O, XFS can delay choosing the physical blocks for newly written data until writeback. This gives the allocator more information about the write and can improve the chance of allocating larger, more contiguous extents.

The allocsize mount option controls the default speculative preallocation size for buffered writes. It can influence allocation behavior, but it is not a guarantee that every write will be allocated in fixed-size chunks or that a checkpoint will consist of only a few extents.

What about Btrfs and ZFS?

Btrfs and ZFS provide valuable features such as snapshots, checksumming, and copy-on-write semantics. Those features can have costs for write-heavy workloads.

For example, when data blocks are shared through reflinks or snapshots, modifying them can require copy-on-write allocation and associated metadata updates. Fragmentation and write amplification can become concerns in some workloads.

However, it is too broad to say that every write on Btrfs or ZFS necessarily causes severe fragmentation or that every transaction causes problematic latency spikes. The outcome depends on workload, record and block sizes, pool or filesystem configuration, and the state of the storage.

There is also an important XFS detail: XFS supports reflink-based copy-on-write when reflink is enabled and extents are shared. Ordinary, unshared file writes are not automatically copy-on-write. If the objective is a scratch filesystem with no need for reflink cloning, decide whether reflink support is useful before formatting the volume.

5. Filesystem comparison for deep-learning workloads

The table below is a qualitative engineering guide, not a universal benchmark result. Actual performance depends on kernel version, filesystem configuration, device topology, dataset layout, and concurrency.

Workload dimension	XFS	ext4	Btrfs / ZFS
High-concurrency metadata activity	Allocation groups can distribute allocation work	Mature performance; contention depends on workload and layout	More complex metadata paths; workload-dependent
Large sequential checkpoint writes	Extents and delayed allocation can help	Generally capable of large sequential writes	CoW and metadata behavior can add overhead in some configurations
Small-file workloads	Capable, but directory and inode behavior should be benchmarked	Often competitive, especially for common directory workloads	Performance varies with metadata and CoW workload
Snapshots and cloning	Reflink support is available when enabled	No native filesystem snapshots or reflink equivalent	Strong native snapshot / CoW feature sets
Direct I/O and GPU-direct paths	Supports direct I/O; GPUDirect Storage depends on the complete platform stack	Supports direct I/O; GPU-direct support is platform-dependent	Support and behavior depend on filesystem and platform
Scratch-storage simplicity	Strong option for large local volumes	Strong general-purpose option	Useful when CoW, snapshots, or checksumming are requirements

Do not interpret this table as proof that XFS is always faster. Benchmark the access pattern that your training job actually uses.

6. The small-file caveat

A common argument against XFS is that ext4 can perform very well with directories containing large numbers of small files. Consider a dataset containing:

• 1.2 million JPEG files, each around 20 KB.
• Several million WAV clips, each around 15 KB.

In this scenario, directory lookup, inode access, path resolution, and file-open overhead can become a significant part of the input pipeline.

But the more important design question is whether millions of individual files should be served directly to a multi-GPU training job in the first place.

Every sample fetched through a loose-file layout may involve operations such as open(), metadata lookup, read(), and close(). At scale, those operations create kernel transitions and metadata-cache pressure that can limit throughput regardless of the filesystem.

Anti-pattern

class LooseAudioDataset(torch.utils.data.Dataset):
    def __getitem__(self, idx):
        path = self.file_paths[idx]
        waveform, sample_rate = torchaudio.load(path)
        return waveform

Better pattern

import webdataset as wds

dataset = (
    wds.WebDataset(
        "/mnt/fast_data/shards/train-{000000..000499}.tar"
    )
    .decode("torch")
    .to_tuple("wav", "cls")
)

Instead of opening a separate file for every sample, the pipeline reads samples from larger sequential containers.

Common options include:

• WebDataset / TAR shards
• HDF5
• Zarr
• Memory-mapped Arrow
• LMDB

A shard size in the range of hundreds of megabytes to a few gigabytes can be a useful starting point, but the right size depends on the number of workers, randomization strategy, storage capacity, and recovery requirements.

Once data is packed into shards, the pipeline can perform larger, more sequential reads. That reduces per-sample metadata overhead and makes it easier to use the bandwidth of fast NVMe storage.

7. Production tuning guide for XFS on NVMe

The following commands are starting points for a dedicated local scratch volume. They are not universal settings. Test them on the target kernel, NVMe device, and workload before deploying them on a production training node.

7.1 Format with an appropriate allocation-group count

For example:

sudo mkfs.xfs -f \
  -d agcount=32 \
  -m reflink=1,crc=1 \
  -b size=4096 \
  /dev/nvme0n1

agcount=32 is an example, not a rule. More allocation groups do not automatically mean more performance. The appropriate count depends on filesystem size, CPU concurrency, and workload. Start with the mkfs.xfs defaults unless you have benchmark evidence that an explicit AG count helps.

The example enables reflink support. If this is a scratch volume and you do not need reflink cloning, consider whether enabling it is necessary. Reflink support does not make every ordinary write CoW, but writes to shared extents do use CoW semantics.

7.2 Mount options

An example /etc/fstab entry:

UUID=xxxx-xxxx-xxxx-xxxx /mnt/fast_data xfs noatime,nodiratime,logbufs=8,logbsize=256k,allocsize=64M,inode64 0 0

Review each option before applying it:

Option	Purpose	Considerations
noatime	Avoids updating file access times on reads	Often useful for read-heavy training data
nodiratime	Avoids directory access-time updates	Generally redundant when noatime is active
logbufs=8	Configures the number of in-memory log buffers	More buffers are not automatically better; benchmark metadata-heavy workloads
logbsize=256k	Sets the log-buffer size	Check the supported range for the running kernel and XFS version
allocsize=64M	Sets the default speculative preallocation size for buffered writes	Test with the checkpoint write pattern; it is not a contiguity guarantee
inode64	Allows inode allocation across the filesystem address space	Commonly appropriate for large filesystems

Do not copy mount options blindly. In particular, options such as logbufs, logbsize, and allocsize should be justified by measurements rather than treated as mandatory tuning.

7.3 Kernel and block-device tuning

NVMe devices use multiple hardware queues. On many modern systems, the none I/O scheduler is a sensible baseline:

# Inspect the active scheduler first
cat /sys/block/nvme0n1/queue/scheduler

# Select the none scheduler
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

For sequential shard reads, you can test a larger read-ahead value:

# Example: 4096 sectors, typically 2 MiB with 512-byte sectors
sudo blockdev --setra 4096 /dev/nvme0n1

You can also experiment with dirty-page thresholds:

sudo sysctl -w vm.dirty_background_ratio=5
sudo sysctl -w vm.dirty_ratio=10

These settings affect system-wide writeback behavior. They can change when dirty pages are flushed and when processes are throttled; they do not guarantee that checkpointing will become faster. Test them under realistic concurrent training and checkpoint workloads, and make any persistent configuration changes deliberately.

8. Diagnosing storage bottlenecks

Before changing filesystems or kernel parameters, establish what the storage path is actually doing.

8.1 Inspect checkpoint extent layout

Use xfs_bmap to inspect the extent map of a checkpoint:

xfs_bmap -v /mnt/fast_data/checkpoints/model_epoch_10.pt

A large number of extents can indicate fragmentation, but extent count alone does not establish that fragmentation is hurting performance. Compare the layout with sequential read and write measurements.

8.2 Monitor device utilization and latency

iostat -xz 1 /dev/nvme0n1

Pay attention to:

• rMB/s and wMB/s: observed throughput.
• await: average I/O wait time.
• aqu-sz: average queue size.
• %util: device utilization as reported by iostat.

A device showing high utilization and low throughput may be handling many small or latency-sensitive requests rather than large sequential transfers. It can also indicate queueing, CPU limitations, or another bottleneck. Compare the results with the device's specifications and with a controlled benchmark; %util alone is not a reliable measure of NVMe saturation.

8.3 Profile CPU-side filesystem activity

Use perf to identify where CPU time is going:

sudo perf top

You can also inspect XFS-related symbols and lock activity where available in your kernel's profiling environment. Symbol names and tracepoint availability vary by kernel, so first inspect the events exposed by the running system rather than assuming a particular event name exists.

The goal is to determine whether the workers are spending meaningful time in filesystem metadata paths, block I/O, decompression, Python overhead, or data transformations.

9. Practical recommendations

For a multi-GPU training node using local NVMe scratch storage:

1. Benchmark XFS and ext4 with the real workload. Include both concurrent sample reads and large checkpoint writes.
2. Prefer sharded datasets over millions of loose files. This often produces a larger improvement than changing filesystems.
3. Use XFS when its allocation-group and large-file behavior fit the workload. Do not assume it wins every small-file or metadata benchmark.
4. Treat mount and kernel parameters as experiments. Change one variable at a time and measure throughput, latency, CPU utilization, and training-step time.
5. Measure the entire pipeline. GPU utilization is an outcome, not a diagnosis. Confirm that storage—not decoding, augmentation, Python, networking, or CPU scheduling—is the limiting factor.

Conclusion

For large local NVMe volumes serving concurrent deep-learning workloads, XFS is a strong filesystem candidate. Its allocation groups can help distribute metadata and allocation work, while extent-based allocation and delayed allocation are useful for large buffered writes.

The biggest gains, however, usually come from treating storage as part of the training architecture. Shard datasets, measure metadata pressure, profile checkpoint behavior, and tune the filesystem against the actual access pattern.

A well-designed input pipeline on a properly benchmarked filesystem is more valuable than a collection of aggressive mount flags. XFS is a good place to start—not a substitute for measurement.