In 2003, three Google engineers — Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung — published a paper describing the storage system they'd built to keep up with Google's data. It's called The Google File System (GFS), and it went on to influence nearly every large-scale storage system that came after it, most directly the Hadoop Distributed File System (HDFS).
This is the first post in a series where I break down classic systems papers that quietly shaped how we build infrastructure today. We'll follow the same structure every time, so you always know what to expect. Today: GFS.
You can read the original paper here: The Google File System (PDF). All credit for the design and ideas below belongs to the original authors and Google.
Problem Statement
By the early 2000s, Google was crawling, indexing, and processing the web — which meant generating and chewing through enormous volumes of data across a lot of cheap machines. They needed a file system that could:
Store many terabytes (eventually petabytes) of data reliably.
Run on inexpensive, commodity hardware rather than expensive specialized storage arrays.
Stay available and correct even though individual machines would constantly fail.
Support workloads that mostly appended data rather than randomly overwriting it — think log files, crawl results, and intermediate computation output.
Serve hundreds of clients concurrently with high throughput, without needing every read or write to be blazingly low-latency.
No off-the-shelf file system at the time was designed around these specific realities. So Google built its own.
Why the Older Approach Struggled
Traditional file systems — including distributed ones like AFS at the time — were built around a different set of assumptions:
Failure was treated as an exception, not a routine event. Systems assumed disks and machines mostly worked, and failure handling was bolted on rather than core to the design.
Files were assumed to be relatively small, and systems optimized for fine-grained I/O rather than huge, multi-gigabyte files.
Random writes were a first-class use case. Filesystems were built to support arbitrary overwrites anywhere in a file, which added complexity that Google's workloads didn't actually need.
Client-side caching of file data was a default assumption, which brings cache-coherence complexity — but Google's workloads mostly streamed through huge files once, so caching data blocks bought little.
POSIX compliance was treated as a requirement, which constrains the API and adds overhead that isn't always earning its keep for a specialized internal system.
None of this made older systems "wrong" — they were solving a different problem. But applied directly to Google's scale and access patterns, these assumptions became a poor fit. So instead of retrofitting an existing file system, Google's engineers designed one from scratch around their actual workload.
Core Architecture
At a high level, a GFS cluster has three kinds of participants: a single master, many chunkservers, and the clients that read and write data.
The Master
One master process holds all the metadata in memory: the file namespace, which files map to which chunks, and where those chunks currently live. Crucially, the master is only involved in control decisions — it tells clients where to find data, but it never touches the actual bytes being read or written. That separation is what keeps it from becoming a bottleneck even as the cluster scales.
Chunkservers
Files are split into fixed-size 64 MB chunks, and each chunk is replicated (three copies, by default) across different chunkservers and different racks. Chunkservers just store chunks as regular files on Linux and serve reads/writes for the byte ranges clients ask for.
Clients
A client asks the master "which chunkserver holds this chunk?", caches that answer briefly, and then talks directly to the relevant chunkserver for the actual data transfer. Metadata traffic and data traffic are cleanly split.
Why 64 MB chunks?
This is one of the more counterintuitive design choices, and it's worth calling out on its own. A large chunk size means:
Clients need to talk to the master far less often, since one chunk covers a lot of data.
The total number of chunks — and therefore the size of the master's in-memory metadata — stays manageable even at huge scale.
Sequential reads and writes, which dominate Google's workload, get to be highly efficient.
Writes and leases
For mutations, the master grants a temporary "lease" to one replica, making it the primary for that chunk. The primary decides the order in which concurrent writes are applied, and the other replicas follow that same order. Data itself is pushed along a pipelined chain between chunkservers — not broadcast from the client to every replica at once — which makes better use of available network bandwidth.
Record append
GFS added an operation not found in traditional file systems: atomic record append. Multiple clients can append to the same file concurrently, and GFS guarantees each individual append lands atomically somewhere in the file — without the clients needing any external locking. This turned out to be a great fit for producer-consumer style workloads, like many worker processes writing results into one shared output file.
Key Trade-offs
GFS makes almost none of its decisions for free — every design win is paired with something given up:
| Choice | What you gain | What you give up |
|---|---|---|
| Single master | Simple design, easy global coordination, easy to reason about | A single point that must be carefully protected (via replication and fast recovery) from becoming a bottleneck or outage source |
| Large (64 MB) chunks | Less master traffic, smaller metadata footprint, efficient sequential I/O | Small files or hot files can create load imbalance ("hot spots") on the few chunkservers that hold them |
| Relaxed consistency model | Simpler, faster implementation; no expensive coordination on every write | Applications must be written defensively — using checksums, unique record IDs, and append-friendly patterns — to tolerate duplicate or padded data |
| No client-side data caching | No cache-coherence complexity | Every read of "hot" data goes back to a chunkserver rather than a local cache |
| Optimized for append-heavy, sequential workloads | Excellent throughput for its target use case | Not a good general-purpose file system — small random writes are supported but never optimized |
The overarching theme: GFS repeatedly chose simplicity and throughput for a specific known workload over generality. That's a defensible trade only because Google controlled both the file system and the applications running on it.
Failure Cases or Limitations
The paper is refreshingly honest about where the system falls short:
Single-client write throughput was disappointing. The paper reports it was roughly half of the theoretical network limit, largely because of how their networking stack interacted with the pipelined replication scheme.
Small files can create hot spots. If a single-chunk file (like an executable pushed to hundreds of machines at once) suddenly gets many concurrent readers, the few chunkservers holding it can get overloaded. Google's real-world fix was operational — higher replication for such files and staggered rollout — rather than architectural.
The relaxed consistency model isn't free for application developers. Because concurrent writes can leave a region "consistent but undefined" (all replicas agree, but the content may be a merge of multiple writers), applications must be written defensively — self-validating records, checksums, and idempotency-friendly identifiers become mandatory, not optional.
The master is a structural bottleneck risk. Even with fast restart and log replication, one process holding all metadata in memory sets a ceiling on file count and requires real engineering (checkpointing, prefix-compressed namespace, binary search over metadata) to keep operations fast at scale.
It is explicitly not POSIX-compliant and not general-purpose. GFS was tuned tightly to Google's own workloads. Bolting it onto a very different access pattern (lots of small random writes, for instance) would likely perform poorly.
What Modern Engineers Can Learn
Even if you'll never build a file system, several ideas from this paper show up constantly in modern distributed systems design:
Design for the workload you actually have, not a hypothetical general one. GFS's biggest wins came from refusing to support use cases (random writes, POSIX semantics) that weren't part of the real workload.
Separate control plane from data plane. The master handles metadata and coordination; chunkservers handle the actual bytes. This pattern — keep the "brain" out of the high-volume data path — recurs everywhere from Kubernetes to modern object stores.
Treat failure as the default state, not an edge case. Fast recovery, replication, and constant background repair aren't bolted on after the fact; they're central to the design from day one.
Relaxing a guarantee can be a legitimate design choice — but only if you push the resulting complexity to a place where it can be handled well (in this case, onto application-level conventions like checksums and idempotent writers).
A single coordinator isn't automatically a scalability bug. If you keep it out of the hot path and make it fast to recover, a centralized component can radically simplify a system without becoming its bottleneck.
How This Maps to AWS / Kubernetes / DevOps
If you work with modern cloud infrastructure, you've already used GFS's descendants, even if the vocabulary is different:
Amazon S3 mirrors the core GFS idea: a durable, replicated, distributed object store optimized for large sequential objects rather than small random-access files, with a metadata layer that's kept separate from the actual data path.
HDFS, the storage layer behind much of the original Hadoop ecosystem, is a near-direct open-source implementation of GFS's ideas: a NameNode (the master) and DataNodes (the chunkservers), large fixed-size blocks, and replication for fault tolerance.
Kubernetes' control plane vs. data plane split echoes the same architectural instinct: the API server and etcd (control plane) coordinate and store metadata/desired state, while the actual application traffic flows directly between pods, never through the control plane.
Kafka's partition replication and leader election resemble GFS's primary/lease mechanism — one replica is elected to order writes for a period, and others follow that order, with automatic re-election on failure.
DevOps takeaway: when you design internal tooling or services, ask the same question GFS's authors asked: what does our actual workload look like, and which "standard" guarantees can we safely relax to buy simplicity and throughput? That question — more than any specific technology — is the real lesson of this paper.
Next in this series: MapReduce (Dean & Ghemawat, 2004).
Source: Ghemawat, S., Gobioff, H., & Leung, S-T. "The Google File System." SOSP '03. Original PDF
Top comments (0)