Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Roughly 720,000 hours of video get uploaded to YouTube every day.
Call it 1,000 terabytes. Tomorrow, another 1,000. The day after, another.
To hold that you need hundreds of thousands of machines with millions of drives spinning inside them.
And here is the uncomfortable arithmetic: in a fleet that size, a drive is dying right now. Another one will die before you finish this post.
Yet not one second of anybody's cat video goes missing.
So how?
The obvious answer is money.
Google is worth a few trillion dollars, so surely they just buy the good computers, the ones that do not break.
Wrong. That computer does not exist.
Physics does not offer an enterprise tier.
Any machine you run will eventually fail, and once you have hundreds of thousands of them, failure stops being an event and becomes a background hum.
That is almost exactly how Google opened the 2003 Google File System paper: component failures are the norm, not the exception.
What YouTube runs on today is a descendant of that design.
The open source clone, HDFS, became the storage layer the entire big data industry stood on for a decade.
Let's build it up from scratch, one broken assumption at a time.
First, the file system you already have
Before we scale to a planet, look at your laptop.
Your operating system ships with a file system whose whole job is organizing bytes on a disk.
It carves your storage into equal sized blocks. Usually 4 KB each.
A 1 TB drive is therefore something like 268 million blocks, numbered from 0 all the way up.
Now save cat.png, a 12 KB masterpiece.
The file system chops it into three 4 KB chunks and drops each chunk into whichever block happens to be free. Not neatly in a row. Wherever there is space.
To ever see your cat again, it records where the pieces went in an index. Every file is a row: here are its chunks, here are the block numbers.
Click the file, the index is read, the chunks are gathered, the cat appears.
Hold that picture, because the rest of this post is the same idea with the blocks replaced by entire computers.
The file that does not fit
Now try to store YouTube.
The biggest enterprise drive you can buy today tops out around a few hundred terabytes and costs about as much as a car you would be nervous to park outside.
One day of uploads already exceeds it.
The naive fix is to build one gigantic machine and jam drives into it until the ingest fits.
Two problems, and neither is subtle.
It is one power cable away from oblivion. One outage, one fire, one clumsy technician, and every video ever uploaded is gone at once.
It does not scale. There is a hard ceiling on how many drives you can hang off a single box, and a much lower ceiling on how many requests it can serve.
So take the local file system idea and stretch it. Instead of many blocks on one machine, use many machines.
Separate boxes, separate storage, separate buildings, separate power, talking over a network.
A file comes in, gets chopped into chunks, each chunk lands on one of those machines.
Call them chunk servers, because they store chunks and later serve them. Each one holds many chunks from many different files.
Then one master plays the role the index table played. It knows every file, its chunk list, and the IP of the chunk server holding each chunk.
A client asks for a cat video. The master hands back a list of chunks and addresses. The client fetches them itself and reassembles the video.
Notice what the master is not doing: it never touches the video bytes. It hands out a map, then gets out of the way.
That single decision is why one master can serve thousands of machines without melting.
The part where everything dies
Now kill a chunk server.
A piece of the cat video just became unreachable, and a video missing a chunk is not a video. It is a buffering spinner with commitment issues.
Your instinct says this is rare. And for one machine your instinct is right. A single server might fail once a year.
But run the numbers across a fleet.
A year is about 31 million seconds. Spread one failure per server per year across a million servers and you get a failure roughly every 30 seconds, forever.
If your system needs every machine online to work, then your system is broken every 30 seconds.
This is the real GFS insight, and it is more philosophical than technical.
You do not build a reliable system by buying reliable parts. You build it by assuming the parts are garbage and designing around their funerals.
Replication buys you time
The first move is the obvious one. Keep more than one copy.
Every chunk is written to multiple chunk servers. Each copy is a replica, and the number of copies is the replication factor, which GFS sets to 3 by default.
The master now tracks, per chunk, the desired replication factor and every server holding a copy.
One server goes dark, the client just asks a different one. No drama.
But be honest about what this actually bought you.
Nothing was solved. Time was purchased.
Run long enough and all three replicas of some chunk will eventually be dead at the same time, and that chunk is gone for good. Three copies of a decaying thing is still a decaying thing.
You need the system to notice the loss and react faster than the losses accumulate.
Heartbeats, and a system that heals itself
Every chunk server sends the master a heartbeat, say every few seconds. It means nothing more than "still here."
Miss a few in a row and the master declares that server dead. It strikes it from the replica list of every chunk it was holding.
Which means those chunks now have two replicas instead of three. Under target.
So the master picks a fresh chunk server that does not already hold that chunk and tells it to copy the chunk from a server that still has a good replica.
Three again.
That loop, running constantly, is the whole trick.
Machines die at a steady rate and re-replication runs at a faster rate, so the system sits in equilibrium while the hardware underneath it quietly rots.
flowchart TD
A[Chunk server heartbeats to master] --> B{Heartbeat received?}
B -->|Yes| C[Mark server alive, refresh chunk map]
C --> A
B -->|No, 3 misses in a row| D[Declare server dead]
D --> E[Remove it from replica list of every chunk it held]
E --> F{Replicas below replication factor?}
F -->|No| A
F -->|Yes| G[Pick a chunk server without this chunk]
G --> H[Copy chunk from a healthy replica]
H --> I[Replica count restored to 3]
I --> A
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef chip fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
classDef bad fill:#ff9a5c,stroke:#c25c1f,color:#1a1a1a
class B,F decision
class A start
class C,G,H,I chip
class D,E badThe nice property here is that nobody pages a human at 3am for a dead disk. The dead disk is a routine input to a loop, not an incident.
But who watches the master?
You have probably spotted the hole.
Every replica of every chunk is tracked by exactly one master, and that master holds the state of the world in its memory.
Congratulations, you built a fleet of disposable machines and then hung its entire availability off one very important box.
The fix rhymes with what you already did.
The master streams every change it makes to a backup master, which sits there receiving updates and doing absolutely nothing else. This is the failover master.
Both masters heartbeat to a health check service.
Clients never hardcode a master IP. They resolve a DNS name, say master.internal.
When the health check service stops hearing from the primary master, it flips that DNS record to point at the failover, which already holds a near current copy of the state and simply takes over.
The pattern repeats at every layer: detect death with heartbeats, keep a warm copy, redirect traffic. Same song, different instrument.
Reading is easy. Writing is where it gets fun.
So far everything has been about pulling data out. Downloads are pleasant because nothing changes underneath you.
Writes are where distributed systems earn their reputation.
Here is the scenario the paper cares about, dressed in something familiar.
You share a spreadsheet with your neighbours for booking apartment parking spots. Each row is a slot. Reserving means appending your name.
Say the whole thing is one chunk, replicated across three chunk servers.
You want the spot. You ask the master for the chunk server addresses, get all three, and send your update to each of them. They confirm. All three copies are identical. Your car is parked. Beautiful.
Now your neighbour wants the same slot at the same moment.
You both get the same three addresses. You both fire off your updates.
Chunk server A receives yours first, then your neighbour's. It appends you, then them.
Chunk server B receives your neighbour's first. It appends them, then you.
The three replicas of a chunk that are supposed to be byte identical now disagree about reality, and there is no way to tell which one is right.
The paper has a word for this. Inconsistent. Data that should be the same on every server is not.
Elect somebody to be right
The root cause is that each chunk server orders updates from its own point of view. It applies what it sees in the order it sees it, cheerfully unaware that its peers saw something else.
Local time is not global truth.
The fix is to stop asking three machines to independently agree and instead appoint one of them to decide. GFS calls that server the primary for the chunk.
The master chooses it, guarantees there is exactly one primary per chunk at any moment, and remembers who it is.
Now the write flow changes shape:
- You and your neighbour both ask the master for the chunk servers, and the master also tells you which one is the primary.
- You both push your data to all three replicas, which buffer it without applying anything yet.
- You both send a write request to the primary.
- The primary picks a serial order for every mutation it received and applies them in that order locally.
- It forwards that same order to the other replicas, which apply it exactly as told, discarding their own opinion about who came first.
Every replica ends up byte identical, no matter how many clients wrote at once.
Data flows to whoever is closest on the network. Order flows from a single point of authority. Separating those two is the elegant bit.
sequenceDiagram
participant You
participant Neighbour
participant M as Master
participant P as Primary replica
participant S as Secondary replicas
You->>M: Where is this chunk?
M-->>You: 3 addresses, plus who is primary
Neighbour->>M: Where is this chunk?
M-->>Neighbour: same 3 addresses, same primary
You->>S: push data (buffered, not applied)
Neighbour->>S: push data (buffered, not applied)
You->>P: write request
Neighbour->>P: write request
P->>P: assign serial order: You, then Neighbour
P->>S: apply in this exact order
S-->>P: applied
P-->>You: done
P-->>Neighbour: doneNote who does not appear in the hot path there. The master hands out a map at the start and then vanishes. All the heavy lifting is client to chunk server.
What GFS deliberately gave up
Every design decision above optimises the same thing: moving enormous amounts of data to enormous numbers of clients.
It is a bandwidth machine, not a latency machine.
Reading a 1 GB chunk stream is glorious. Reading one 200 byte record with a tight deadline is not what this was built for, and the designers knew it.
That is the actual lesson hiding inside GFS, and it generalises far beyond storage.
You pick the two or three properties your system genuinely needs, you optimise those relentlessly, and you take the others off the table on purpose.
A system that refuses to choose is a system that is mediocre at everything.
A question to sit with
Here is the one the paper leaves you with, and it is worth chewing on before you look it up.
Imagine one file in your GFS cluster goes viral.
A single post, a single video, and suddenly a large slice of all traffic is hammering the handful of chunk servers holding its chunks.
Those machines are drowning while the rest of the fleet is idle. The paper calls this a hotspot.
How would you spread that load?
The chunk size section of the paper has the answer Google reached for, and it is a smaller change than you would expect.
Go read it. It is fifteen pages, it is written in plain English, and it is one of the few papers that reads like somebody explaining a thing they actually built rather than a thing they wanted funded.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score |
|---|---|---|
 |
 |
 |
How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)