Making a remote file look ordinary is easy at the API boundary. Keeping that illusion intact through latency, caching, concurrent access, and server failure is where the real filesystem begins.
"Distributed file systems make remote storage feel local, but latency, caching, consistency, retries, and server failures reveal the complexity underneath. This article explores NFS, VFS, file handles, caching, stateless recovery, and the trade-offs behind filesystem transparency."
Local files have trained us well.
Open a file.
Read some bytes.
Write some bytes.
Close it.
fd = open("/work/report.txt")
read(fd, buffer, n)
write(fd, buffer, n)
close(fd)
Underneath that tiny interface, the filesystem is doing ✨a lot✨
Disk blocks have to become files.
Names have to map to those files.
Permissions have to be checked.
Data has to survive failures.
Directories, i-nodes, allocation, buffering, and storage layout all exist below the API.
But the application mostly gets to ignore them.
That is the beauty of the filesystem abstraction.
So when the file moves onto another computer, the natural ambition is obvious:
Keep the abstraction.
Why should an application care whether /work/report.txt lives on the local disk or on a server across the network?
Let it call open , read , and write exactly as before.
The distributed file system can deal with the awkward details underneath.
This is a very good idea.
It is also where the trouble starts.
The First Goal Is to Make Distance Disappear
A distributed file system allows processes on different machines to share persistent files.
Ideally, accessing a remote file should feel much like accessing a local one.
That requirement immediately creates several forms of transparency.
Access transparency says the application should not need one API for local files and another for remote files.
Location transparency says the pathname should not force the application to care where the file is physically stored.
Mobility transparency says moving a file should not require every client to be rewritten.
Performance transparency asks the service to remain useful as load changes.
Scaling transparency asks it to grow without changing the way applications interact with it.
This is an ambitious little collection of promises.
The application wants: read(file)
The system underneath may need:
- identify remote filesystem
- resolve pathname
- find server
- construct RPC
- send request
- wait across network
- authenticate user
- locate file
- read storage
- send data back
- update cache
- return bytes
That complexity is not a reason to abandon transparency.
It is the reason transparency is valuable.
The application should not have to become a distributed-systems expert merely because somebody moved its file.
So the architecture begins building an illusion.
Give the Client One Interface and Hide the Rest
A useful distributed file service can be separated into responsibilities.
A flat file service operates on file contents and attributes using unique file identifiers.
A directory service maps human-readable names onto those identifiers.
A client module combines those services behind the interface applications actually use.
That separation solves an important problem.
Names and files are not the same thing.
/projects/report.txt is useful to a human.
The system still needs an identifier that tells it which actual file is being referenced.
The client module becomes the diplomat between those worlds.
It knows where remote file and directory services live.
It can translate local-looking operations into distributed requests.
And, importantly, it can maintain a cache of recently used data.
That last feature will become both hero and villain shortly.
But first we need to make the remote filesystem look local.
NFS gives us a particularly clean example.
VFS Keeps Up the Act
In the Network File System, the client does not ask applications to switch personalities when a pathname crosses the network.
The client operating system uses a Virtual File System , or VFS, layer.
The application makes familiar UNIX filesystem calls.
VFS determines what kind of filesystem the request belongs to.
Local file?
Handle it through the local filesystem.
Remote NFS file?
Pass it toward the NFS client.
The NFS client translates the request into NFS protocol operations and communicates with the server.
NFS version 3 itself is defined as a remote procedure call protocol for accessing files on a server. (rfc-editor.org)
So the application sees: read(fd, buffer, 4096)
while the NFS client may be performing an RPC against another machine.
That is access transparency doing its job.
The application does not need to care.
The kernel very much does.
A Remote File Needs More Than an i-node
On a local UNIX filesystem, an i-node can identify the filesystem object.
But an i-node number by itself is not enough when another machine has to refer to that object.
The NFS server therefore gives clients an opaque file handle.
Conceptually, that handle contains enough information for the server to locate the correct file again.
It can identify the filesystem.
The file within that filesystem.
And even help distinguish a current file from an older object whose i-node number has since been reused.
The client does not need to interpret the handle.
That is why it is opaque.
The server is effectively saying:
Keep this token. Hand it back when you want to talk about this file again.
VFS can then keep information for mounted filesystems and opened files, distinguishing local objects from remote ones while presenting them through the same application-facing interface.
The local illusion is getting convincing.
Now we attach the remote filesystem into the local directory tree.
Mounting Moves the Boundary Without Moving the API
Suppose a server exports: /export/people
A client mounts it at: /usr/students
From the client’s point of view, the remote tree now appears inside its ordinary local namespace.
An application walks through: /usr/students/alice/report.txt
It does not need to know that part of that pathname suddenly crosses onto another machine.
The mount operation established the boundary.
VFS remembers it.
When pathname translation reaches that mount point, subsequent operations can be translated into NFS requests using the corresponding server information and file handles.
This is location transparency working remarkably well.
The pathname looks continuous.
The machines underneath are not.
And this is where it becomes tempting to forget the network entirely.
After all, if the interface is the same and the pathname is the same, perhaps the behaviour should be the same too.
Then somebody reads the file.
The Network Charges Rent on Every Read
A local filesystem can fetch a block from local storage.
A remote filesystem may have to send a request across the network, wait for the server, let the server retrieve the data, then move that data back across the network.
Do that for every read and the illusion gets expensive very quickly.
Latency has entered the filesystem.
So the obvious solution is the same one computing keeps rediscovering whenever something far away is slow:
Cache it closer.
Keep recently used blocks at the client.
Now repeated reads can often avoid contacting the server.
Read-ahead can fetch the next likely block before the application asks for it.
Writes can sometimes be delayed rather than forcing the application to wait for every remote storage operation.
Performance improves.
The remote file starts feeling local again.
Excellent.
We have also quietly created another copy of the file’s state.
That is where the cache becomes confidently stale.
Caching Solves the Distance Problem by Creating a Consistency Problem
Suppose Alice and Bob both access the same remote file.
Alice reads block X.
Her client caches it.
Bob modifies block X through another client.
The server now has the newer version.
Alice’s machine still has the old one sitting pleasantly in memory.
Alice reads again.
What should she see?
On a conventional single-copy filesystem, the expectation is intuitive:
If a write happens and a later read observes that file, the read should see the updated value.
But distributed caching means multiple copies exist at different locations.
Updates take time to propagate.
The system has improved performance by weakening the assumption that every read necessarily consults the one authoritative copy.
NFS version 3 explicitly acknowledges this trade-off: it permits client caching but does not provide strict cache consistency between client and server or among different client caches.
This is the uncomfortable part.
Caching was not a bad optimization.
Without it, remote access could become painfully dependent on network latency.
But once we cache, one-copy semantics becomes expensive.
The copies need some way to decide whether they are still trustworthy.
The Cache Has to Periodically Doubt Itself
One practical strategy is timestamp-based validation.
A client remembers when a cached object was last validated.
It also remembers modification information obtained from the server.
For a limited freshness interval, the client can assume the cached entry remains usable without contacting the server.
After that interval, confidence expires.
The client asks the server for current attributes.
If the server’s modification information still matches what the client remembers, the cached data survives another round.
If it differs, the client knows its copy has become stale and must fetch fresh data.
This is a lovely compromise.
The client does not contact the server on every read.
That would defeat much of the point of caching.
It also does not trust the cache forever.
That would make inconsistency unbounded.
The cache behaves like an engineer who has learned not to be too confident:
I checked recently. Probably fine.
Then:
It has been a while. Better ask.
NFSv4 continues to make cache validation part of the protocol model; cached data and names have to be revalidated under defined conditions because concurrent clients can change the underlying files.
The system did not eliminate inconsistency.
It bounded and managed it.
Writes Make the Trade-Off Meaner
Reads are comparatively polite.
Writes change authority.
Suppose the client modifies a cached block.
The cache now contains data the server may not yet possess.
The modified page becomes dirty.
One approach is write-through caching.
Every update is written through to stable server storage before the operation is considered complete.
That gives stronger durability semantics.
It also means every write potentially waits for remote work.
Not especially charming if writes are frequent.
So another strategy allows writes to reach server memory first and become persistent later through a COMMIT operation.
NFS version 3 explicitly supports this distinction. A client can issue unstable writes and later use COMMIT to force previously written data to stable storage; until that happens, the client must be prepared to retransmit the data if the server loses its volatile state.
Now “write completed” itself has acquired layers.
Copied into the client cache.
Sent to server.
Stored in server memory.
Committed to stable storage.
Once again, the local API is hiding a lifecycle the distributed implementation cannot ignore.
Then the Server Dies
Suppose the application has a file open.
The NFS server crashes.
This sounds like the moment where everything should become dramatically complicated.
Oddly, one of NFS’s most interesting design choices exists specifically to make this recovery less complicated.
NFS version 3 assumes a stateless server.
The server does not need to remember a client’s open-file session in order to process ordinary filesystem requests correctly.
The client sends enough information with each request — including the file handle and operation parameters — for the server to understand what should be done.
Why refuse to remember useful information?
Because remembered state creates recovery obligations.
Imagine the server kept essential information saying:
Alice opened this file.
Her current position is here.
She owns this session.
Then the server crashes.
That state disappears.
After reboot, reconstructing the world may require recovering every client’s session correctly before ordinary requests can continue.
A stateless design gives the server fewer ghosts to recover.
The NFSv3 specification makes this motivation explicit: because correctness does not depend on per-client server state, a crashed server can restart and clients can retry their requests rather than reconstructing an essential session database.
NFS made recovery easier by deliberately remembering less.
That feels backward until failure enters the design.
Then it feels inevitable.
Retrying Only Works If Repeating Is Safe
Stateless recovery sounds almost magical.
Server disappears.
Server returns.
Client retries.
Carry on.
There is one small problem.
What if the original request actually reached the server, but the reply did not reach the client?
The client cannot necessarily tell.
So it retries.
Now the server may receive the same logical operation twice.
This should feel familiar.
The same ambiguity appeared with remote procedure calls because NFS is built on remote invocation.
The distributed file service therefore has to think carefully about invocation semantics.
With at-most-once semantics, the system tries to avoid executing duplicated requests.
Another approach permits at-least-once invocation but designs operations to be idempotent where possible.
An idempotent operation can be performed repeatedly without repeated execution changing the final result beyond the first successful execution.
That property makes retries far less terrifying.
The retry logic is not there merely because networks are unreliable.
It exists because a filesystem operation has crossed a boundary where the client may lose knowledge of whether the server performed it.
Once again, the local-looking API hides a distributed uncertainty underneath.
What Should a Failed Server Feel Like?
Now we reach one of the most wonderfully uncomfortable NFS choices.
The client makes a request.
The server does not answer.
What should happen to the application?
A hard-mounted filesystem keeps retrying.
The process attempting to access the remote file may remain suspended until the server becomes reachable again.
From the application’s point of view, the operation appears to hang.
That sounds awful.
But notice what the system is protecting.
A transient network failure does not suddenly turn an ordinary filesystem operation into an unexpected I/O failure that the application may not know how to handle.
The filesystem keeps trying to preserve the illusion that the remote storage still exists.
A soft-mounted filesystem chooses differently.
After a limited number of retries, it gives up and returns an error to the application.
Now the application regains control.
But the network failure has leaked through the filesystem abstraction.
Oracle’s NFS documentation describes the trade-off directly: hard mounts continue retrying until the server responds, whereas soft mounts return an error after giving up; it also warns that applications may handle such soft-mount failures poorly.
Neither behaviour is magically correct.
Hard mount says:
I would rather make you wait than pretend your file operation failed permanently.
Soft mount says:
I would rather expose the failure than make you wait indefinitely.
This is where the title earns itself.
The file looked local.
Then the network disappeared.
Now the system has to decide how much of that disappearance you are allowed to notice.
Transparency Has a Price Tag
Look back at what it took to make a remote file appear ordinary.
VFS hid the distinction between local and remote access.
Mounting inserted a remote filesystem into the local namespace.
File handles gave the server a way to identify remote objects.
RPC carried filesystem operations over the network.
Statelessness made server recovery simpler.
Retries helped tolerate transient communication failures.
Caching hid latency.
Read-ahead tried to make future access cheaper.
Delayed writes prevented every modification from immediately blocking on the server.
Validation mechanisms tried to stop cached copies from becoming confidently wrong.
Locking became necessary when concurrent clients modified shared files.
Replication could improve availability and distribute load.
Every mechanism protects part of the illusion.
Every mechanism also introduces another trade-off.
Caching improves performance but complicates consistency.
Replication improves availability but creates more copies to coordinate.
Strong concurrency control protects correctness but costs communication and synchronization.
Statelessness simplifies recovery but constrains how the protocol is designed.
Hard mounts hide transient failure but can leave applications waiting.
Soft mounts expose failure but force applications to deal with semantics they may have assumed the filesystem would hide.
Distributed filesystems are not simply local filesystems plus networking.
The network changes what the filesystem can promise.
The File Was Never Really Local
That does not make the abstraction dishonest.
Quite the opposite.
A good distributed file system hides details that applications should not need to repeat.
Most programs should not manually locate servers.
They should not construct RPCs for every block read.
They should not maintain their own file-handle translation.
They should not each invent cache validation.
They should not each reconstruct server-recovery protocols.
The filesystem abstraction absorbs that complexity so applications can continue saying:
- open
- read
- write
- close
But there is a line between hiding implementation detail and pretending physics changed.
Remote storage still has latency.
Communication can still fail.
Servers can still disappear.
Caches can still become stale.
Replicas can still disagree.
Concurrent writers still need coordination.
Persistence still has to reach stable storage eventually.
The Origami Software Engineer still has to deal with it.
The pathname cannot make those constraints disappear.
It can only give us a cleaner place to manage them.
That is the deeper trick behind distributed file systems.
The goal was never to make a remote disk physically equivalent to a local one.
The goal was to preserve the useful filesystem abstraction while designing explicitly for everything that becomes different once storage crosses a network boundary.
The file looked local because a remarkable amount of machinery worked to make it feel that way.
Then the network disappeared and reminded us where the file really was.
That’s not failure.
That’s evolution.
The “I liked this” Starter Pack:
Don’t let your fingers get lazy now.
- Like : It tells me this was worth writing.
- A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
- Boost it: Especially with that one developer who definitely needs this.
Thanks for being here. It genuinely helps more than you know!
— Aaroophan Varatharajan
Find me elsewhere:
- Professional stuff: linkedin.com/in/Aaroophan
- Code stuff: github.com/Aaroophan
- UI stuff: aaroophan.dev/Aaroophan
- Life stuff: instagram.com/Aaroophan
Top comments (0)