The classic Linux answer to "what's changing in that directory?" is inotify. It works beautifully on small trees; the problem is that watching is set up per directory. When you want to watch a storage area with hundreds of thousands of directories, you need a watch for each one — and at some point you hit the limit.
Here's the limit on my own server:
$ sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
fs.inotify.max_user_watches = 753024
fs.inotify.max_user_instances = 128
Seven hundred thousand sounds like plenty, but not for someone running a file-sharing server — and long before you reach that number, walking the tree to mark every directory becomes a job that takes minutes on its own.
fanotify asks a different question: instead of marking directories one by one, what if we watched an entire mount point or filesystem? And it does one more thing — it can not only see access but stop it.
The thesis of this article: what makes fanotify interesting isn't the scale, it's that second capability. And that capability carries a serious responsibility: if your listener doesn't answer, the process trying to open the file waits.
Two APIs, two mindsets
With inotify the model is simple: you give a path, you get events about that path. With fanotify a mark has three scopes, and the choice shapes your architecture: individual files and directories, a mount point with FAN_MARK_MOUNT, or an entire filesystem with FAN_MARK_FILESYSTEM.
The privilege side deserves precision, because most write-ups are sloppy here. fanotify isn't entirely a root affair: since 5.13 the FAN_CLASS_NOTIF class can be opened by an unprivileged user, provided one of the file-handle reporting flags is set. The comment in the kernel source explains why — unprivileged groups don't get file descriptors in events, so handle reporting is mandatory.
But here's the crux: the wide-scope marking that sells this API isn't available unprivileged. Both FAN_MARK_MOUNT and FAN_MARK_FILESYSTEM require CAP_SYS_ADMIN; an unprivileged group can only place individual inode marks, sees no pid or file descriptor in events, and can't use the unlimited-queue or unlimited-marks flags. So "one mark covers the whole disk" isn't free — it comes with root.
That scope difference is the difference between "I'll watch 10 directories" and "I'll see what happens on this disk". The second is theoretically possible with inotify but means walking the tree and opening hundreds of thousands of watches; with fanotify it's a single call.
The resource limits became tunable in 5.13. The values on my server:
$ sysctl fs.fanotify.max_queued_events fs.fanotify.max_user_groups fs.fanotify.max_user_marks
fs.fanotify.max_queued_events = 16384
fs.fanotify.max_user_groups = 128
fs.fanotify.max_user_marks = 800683
These three cap the number of queued events, the number of groups per user, and the number of marks per user respectively. Before 5.13 they were hardcoded constants: 16384 events, 128 groups per user, and 8192 marks per group.
You may have noticed something: fanotify's mark limit (800683) is only 6% above inotify's watch limit (753024). The gain isn't "more marks you can place"; the gain is that a single mark covers an entire tree.
The real difference: the power to permit
When you initialise an fanotify group you choose a "notification class", and that choice determines what you can do.
FAN_CLASS_NOTIF is the default: you're told that a file was accessed, but you can't decide before the access happens. That's the class for accounting and auditing.
FAN_CLASS_CONTENT and FAN_CLASS_PRE_CONTENT unlock permission events. The man page even names the use cases: the first is for listeners that need to access files once they already contain their final content — malware detection programs are the example given; the second is for listeners that may need to write data before the final data can be accessed — hierarchical storage managers are the example. The two permission classes require CAP_SYS_ADMIN; FAN_CLASS_NOTIF can be opened unprivileged under the conditions described above.
With a permission class you receive FAN_OPEN_PERM, FAN_ACCESS_PERM and FAN_OPEN_EXEC_PERM (open with intent to execute) events, and for each one you must write a response: FAN_ALLOW grants, FAN_DENY refuses. A denied call receives EPERM. Since Linux 6.13, groups initialised with FAN_CLASS_PRE_CONTENT can deny with a different error using FAN_DENY_ERRNO() — you can make the application see EIO, for instance. It isn't an arbitrary error code though; the man page lists the permitted set: EPERM, EIO, EBUSY, ETXTBSY, EAGAIN, ENOSPC, EDQUOT.
The last two boxes are what anyone considering fanotify in production needs to digest first — and this is where intuition fails.
If your listener is alive but not responding (a hung thread, an unread queue), processes trying to reach the file wait; those waiting tasks are killable, so SIGKILL gets you out. But if the listener dies, the picture inverts: the man page is explicit — upon close, outstanding permission events are set to allowed. The kernel source does the same thing; as the group is released, every pending permission event is finished with FAN_ALLOW.
So the real risk isn't a wedged system, it's your security gate opening silently. When a listener doing malware scanning crashes, everything looks normal; there's simply nobody at the door any more. There's an interesting asymmetry too: when memory can't be allocated the kernel denies the permission event, failing closed in that case.
Linux 6.18 shed a little light on this blind spot: with fs.fanotify.watchdog_timeout, if a permission event goes unanswered for the configured period the kernel logs a warning naming the PID that failed to respond to the queue. It doesn't resolve the event, but it does make "the doorkeeper fell asleep" visible.
Lost events and an overflowing queue
The most dangerous state for a monitoring system is silently missing events while believing it sees everything. fanotify is honest here: when the queue limit is exceeded a FAN_Q_OVERFLOW event is generated. There's one exception, and it concerns exactly the permission class: overflow events aren't queued for permission events — there, access is denied outright. The FAN_UNLIMITED_QUEUE flag removes the limit — and again requires CAP_SYS_ADMIN.
One more detail: events can be merged. Consecutive events for the same filesystem object originating from the same process may be merged into a single event. The one exception is permission events — two of those are never merged into one queue entry.
The analytical consequence is clear: fanotify doesn't give you a "how many times was it read" counter, it gives you "it was read". If you're counting, account for merging.
Choosing between descriptor and handle
In classic usage each event carries an open file descriptor, and closing it is the reading application's responsibility. In a high-volume system that's the easiest path to descriptor leaks.
The alternative is initialising the group with flags like FAN_REPORT_FID or FAN_REPORT_DIR_FID. But know this first: the trade-off only exists in the notification class. The kernel rejects fid mode together with any class other than FAN_CLASS_NOTIF with EINVAL. In a permission class there is no choice — every event carries a descriptor and you must close it. For anyone building a permission gate, descriptor-leak risk isn't a preference, it's a given. Objects are then identified by file handles instead of descriptors, and a separate information record arrives with the event. These flags stack; with FAN_REPORT_TARGET_FID and FAN_REPORT_PIDFD together, one event can carry two fid records and a pidfd record. The man page's warning matters: with such a stacked configuration there is no guarantee about the ordering of information records.
If you want to turn a handle into a path, open_by_handle_at() comes into play — the answer to "which file was it" isn't free, it's an extra call. In exchange you don't open and close a descriptor for every event; at high volume that trade usually favours handles.
In practice: use a descriptor if you're going to read the file's contents, and a handle if you only need to know which object it was.
The things it doesn't see
The man page's "limitations and caveats" section is short but it shapes production decisions.
fanotify reports only events that a user-space program triggers through the filesystem API, so it doesn't catch remote events on network filesystems. A change made by another client on a directory you share over NFS never reaches your listener.
Accesses and modifications happening through mmap(), msync() and munmap() are not reported either. If you're watching an application that works with memory-mapped files, the picture you see is incomplete.
Events for directories are created only if the directory itself is opened, read and closed; adding, removing or changing children of a marked directory doesn't create such an event. To see changes in directory contents you have to request separate event types like FAN_CREATE and FAN_DELETE. There's a scope trap here as well: these handle-identified events can't be supplied in the mask together with FAN_MARK_MOUNT — the attempt returns EINVAL. So the plan of "mark the mount point and watch what gets created on the file server" doesn't work directly.
There's also a security note that bears directly on the permission class: the kernel doesn't check whether the receiving process is authorised to read or write the file before passing it a descriptor. When CAP_SYS_ADMIN is granted to programs run by unprivileged users, that's a risk.
Locking yourself out
There's a trap every listener author in a permission class eventually hits, and it's easy to miss in the man page: the descriptor delivered with the event has FMODE_NONOTIFY set, so access through that descriptor generates no new events.
But if your listener reopens the file by path — while writing a log, copying to quarantine, or resolving a handle with open_by_handle_at() — it triggers its own permission event and blocks itself. That's the classic fatal bug in fanotify-based scanners. The rule is simple: while making the decision, use only the descriptor you were given, and keep your own open() calls away from the path you're gating.
A second detail: marks aren't retroactive. Access through descriptors opened before your listener started is invisible to you. Service ordering is therefore a security matter.
The state on my server
Two kernel options decide, and both are on for me:
$ grep CONFIG_FANOTIFY /boot/config-$(uname -r)
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
The first enables the API, the second the permission handling. On a kernel with the second one off, trying to open a FAN_CLASS_CONTENT group goes nowhere — don't start a permission-based design before verifying this on your distribution's kernel.
What changed in recent releases
fanotify isn't a static API; several recent additions touch this article's topics directly.
FAN_MARK_EVICTABLE (5.19) belongs to the scale side: it doesn't pin marked inodes in memory, lowering the memory cost on wide trees. FAN_MARK_IGNORE (6.0) is the modern replacement for the old ignore mask and the right way to narrow an event flood. FAN_FS_ERROR (5.16) surfaces filesystem errors and FAN_RENAME (5.17) renames as their own events.
6.14 advanced on two fronts: mount namespace watching (FAN_MARK_MNTNS plus mount attach/detach events) and genuine pre-content events (FAN_PRE_ACCESS) — meaning the hierarchical storage manager scenario is no longer just a theoretical example in the man page.
When fanotify, and when not?
The question for your own setup isn't "which API is more powerful" but "what am I going to do".
If you're watching a handful of configuration files to reload a service, inotify is enough; there's no need to pay fanotify's CAP_SYS_ADMIN price. If you're on a file server working out access patterns, answering "which share is actually used", or producing an audit trail, fanotify's notification class is the right tool: one mark covers an entire mount point.
If you want to block access — malware scanning, a data classification gate, "this file must not be opened before it's restored from the archive" — the permission class is the only way. But then your listener is an infrastructure component: restarts, timeouts, crash scenarios and the question "what should happen when the listener isn't there" all become part of your design.
If an audit trail is your actual goal, don't decide without comparing fanotify to auditd: auditd works at the system call level and gives richer user/process context, while fanotify works at the filesystem object level and comes with the ability to block. The rule structure in my runbook on monitoring privileged commands with Linux auditd answers "who ran what"; fanotify is strong on "which file was touched". Nor are they entirely separate worlds: open the fanotify group with FAN_ENABLE_AUDIT and set the FAN_AUDIT flag on responses, and your permission decisions land directly in the audit subsystem. You'll be shipping whatever you collect somewhere, which is a design job of its own.
One more warning: file access records are inherently sensitive data. Which user opened which document is, in most organisations, more critical than the log itself. Decide on retention and access rights before you start collecting.
Checklist before you build
- Verify kernel support:
CONFIG_FANOTIFY, andCONFIG_FANOTIFY_ACCESS_PERMISSIONSif you need permissions. - Pick the scope: file/directory,
FAN_MARK_MOUNTorFAN_MARK_FILESYSTEM. The wrong scope means either far too many events or missing coverage. - Size
fs.fanotify.max_queued_eventsfor your workload and always handleFAN_Q_OVERFLOW— a listener that misses overflow has incomplete data. - If you use descriptors, put the close on every branch of the code; if you don't, move to the
FAN_REPORT_FIDfamily. - In a permission class, test timeouts and the "listener died" scenario; run your first production attempt on a narrow mount point, not an entire filesystem.
- Document the blind spots — mmap, network filesystems, directory child events — so nobody has to ask later why something wasn't seen.
The difference between watching and deciding
What strikes me most about fanotify is that both capabilities live in the same API. Watching is passive: get it wrong and you collect incomplete data. Deciding is active: get it wrong and you stop the system.
Because they sit behind the same interface, the two get conflated easily — "we're watching anyway, let's block too" sounds so innocent. Yet the moment you start blocking, your listener becomes a component as critical as your database.
So the question for your own setup: do you really want to decide, or only to see? If it's the latter, stay in the notification class.
Top comments (0)