For most of its history, every read and write on a ZFS pool passed through the Adaptive Replacement Cache, the in-memory cache that gives ZFS much of its responsiveness. OpenZFS 2.3 changed that by introducing Direct I/O, and OpenZFS 2.4, released in December 2025, refined it with an uncached I/O fallback. ZFS Direct I/O lets suitable requests skip the ARC entirely. On fast NVMe pools that can be a real win, but switching it on without understanding the trade-offs can just as easily make a NAS slower.
What the ARC Normally Does for You
The ARC holds recently and frequently used data in RAM, balancing between the two so that a single large scan does not evict everything useful. Reads served from memory avoid disk entirely, and writes are buffered and aggregated into transaction groups before being committed, which lets ZFS lay data out efficiently and compute checksums in batches.
That design made sense when disks were slow and memory was comparatively fast. With modern NVMe arrays, the gap has narrowed. Copying every byte through the cache costs memory bandwidth and CPU time, and for data that will never be read again, caching it simply evicts more valuable blocks.
How Direct Requests Take the Short Path
Applications request direct access by opening files with the O_DIRECT flag, a convention long used by databases and some virtualization and backup tools. With the feature enabled, OpenZFS can service those requests by moving data between the application buffer and the pool without keeping a copy in the ARC. Checksums, compression and the rest of the integrity pipeline still apply.
Administrators control behavior per dataset with the direct property. The default honors applications that ask for O_DIRECT, an "always" setting treats eligible I/O as direct regardless of flags, and a "disabled" setting ignores the request and uses the cache as before. Alignment matters: requests must match block and page boundaries to qualify. In OpenZFS 2.4, misaligned direct requests can fall back to an uncached path rather than filling the ARC, which reduces surprise cache pollution.
It helps to be clear about what ZFS Direct I/O does not change. Data is still written in transaction groups, snapshots and clones behave exactly as before, and replication with send and receive is unaffected. Block cloning and native encryption continue to work at the pool level. The feature alters where data travels on its way to and from the disks, not how ZFS protects or organizes it once it is there. That makes it relatively low risk to trial on a single dataset, because disabling the property returns the dataset to cached behavior without any migration or rewrite of existing data.
Workloads That Gain From Skipping the Cache
The clearest winners are workloads that already manage their own caching or never reread data:
- Databases with their own buffer pools, where the ARC would hold a second, redundant copy.
- Large sequential streaming writes, such as media ingest or scientific output, on fast NVMe pools.
- Backup and archive jobs that write data once and read it rarely, if ever.
- Analytics scans over datasets far larger than available memory.
For these, ZFS Direct I/O frees memory for data that benefits from caching and can reduce CPU overhead per byte. Large pools built for scale-out NAS workloads such as IoT and big data often see this pattern of write-once, scan-occasionally data.
Where Bypassing the ARC Backfires
General file sharing is a poor fit. Office documents, home directories and project folders are read repeatedly by many users, and the ARC is exactly what keeps those reads fast. Setting a busy SMB or NFS share to "always" can turn memory hits into device reads and raise latency noticeably.
Spinning-disk pools are another risk. Hard drives rely heavily on caching and write aggregation to hide seek time; removing that layer exposes their mechanical limits. Small, random or misaligned writes also gain little, because they may not qualify for the direct path in the first place.
Interactions With Recordsize, Compression and Sync
Recordsize shapes how well direct requests align. An application issuing large aligned writes to a dataset with a matching recordsize has the best chance of taking the direct path; a mismatch pushes more I/O onto fallback paths. Compression and checksumming still run, so CPU cost does not disappear, it just avoids the extra memory copy.
Sync semantics still apply as well. Applications that demand synchronous writes continue to rely on the ZFS intent log, so a fast log device remains relevant. The protocol layer matters too; block access over iSCSI and file access over NFS or SMB behave differently, a distinction explored in this overview of how SAN, NAS and DAS storage differ.
Testing Before Flipping the Property
Change the setting on one dataset at a time and measure with the real application, not a synthetic tool alone. Watch ARC hit ratios, memory pressure, CPU utilization and tail latency before and after. A rising hit ratio on other datasets, freed by fewer cache evictions, is often the most visible benefit.
Backup repositories are a sensible first candidate because their access pattern is predictable. If your pool serves as a target for image-based backups, the setup guidance in this article on building a Veeam backup repository on NAS pairs well with a direct-I/O test on that dataset.
Choosing Per Dataset, Not Per Pool
ZFS Direct I/O is a precise tool rather than a global performance switch. Leave the default in place for general shares, consider "always" only for datasets holding self-caching databases or write-once streams on fast flash, and verify results with real workloads. Used selectively, it lets the ARC concentrate on the data that actually benefits from memory, which is where ZFS performance has always come from.
Top comments (0)