Every request your code sends across an ocean — every API call to a region on
another continent, every git pull from a mirror overseas — travels through a
fiber optic strand roughly the width of a garden hose, resting on the seabed.
When one of those strands snaps, and they snap about two hundred times a year,
the fix depends on a fleet so small and so old that a recent piece explaining
how sixty old ships are holding up the internet
reads less like trivia and more like an incident report for civilization
itself. As developers, we obsess over software resilience — retries, circuit
breakers, multi-region failover — while the physical layer underneath all of
it runs on hardware that would fail any audit we'd ever write for our own
systems.
The Physical Layer Nobody Budgets For
Here is the part that surprises most engineers: there is no global authority
responsible for fixing broken internet cables. Repairs are handled by private
maintenance agreements between cable owners and a handful of ship operators,
a structure that has barely changed since the telegraph era. Josh Dzieza's
extraordinary reporting in
The Verge's deep dive into the subsea repair industry
found that of the several dozen cable ships in existence worldwide, only a
fraction are dedicated to repair work, many are past forty years of service,
and the veteran crews who know how to operate them are retiring faster than
replacements can be trained.
Think about what that means in engineering terms. The global network carries
an estimated 99 percent of intercontinental data traffic through this
infrastructure. The repair capacity — the thing that determines your mean
time to recovery when a cable fails — is a bottleneck with no redundancy
plan, no on-call rotation deep enough, and a hardware refresh cycle measured
in decades. If a service you ran had a recovery path this thin, you would
escalate it as a critical risk before lunch.
How a Repair Actually Works
The process itself is almost absurdly analog. When a fault is detected,
operators use signal testing to estimate its location, sometimes within a few
hundred meters across thousands of kilometers of cable. A ship then sails to
the coordinates and, in deep water, drags a grapnel hook — a technique
essentially unchanged since the 1860s — along the seabed to snag the cable.
The damaged section is hauled up, cut out, and a new segment is spliced in by
hand, fiber by fiber, in a clean room aboard the ship. A single repair
typically takes one to two weeks, assuming a ship is available, the weather
cooperates, and the fault isn't in contested or dangerous waters.
Now multiply the problem. Earthquakes, dragged anchors, and fishing gear
don't break one cable politely; they often take out several at once. After
the 2011 Tōhoku earthquake, a single Japanese ship spent months at sea
repairing fault after fault while radiation from Fukushima complicated every
operation. The system held — but it held because of individual expertise, not
because of structural resilience.
Why This Suddenly Matters More
The stakes have shifted from accidental damage to something darker. A string
of cable failures in the Baltic Sea in late 2024 triggered sabotage
investigations across several European governments, and as
IEEE Spectrum's analysis of protecting undersea cables
makes clear, defending hundreds of thousands of miles of exposed fiber
against deliberate interference is close to technically impossible. You
cannot patrol the entire ocean floor. Satellite backup exists mostly as
proof-of-concept work, years away from carrying meaningful load. The honest
answer from the industry is the same one distributed-systems engineers
already know: you don't protect every node, you build enough redundancy that
losing several doesn't matter.
Which is exactly why the aging repair fleet is the real story. Redundancy
only works if broken paths come back online faster than new failures
accumulate. Fewer ships, older ships, and retiring specialists all push
recovery times in the wrong direction while cable count — driven by
hyperscalers laying private systems — keeps climbing.
What Developers Can Take From This
There are a few practical lessons worth internalizing from all of this:
- Latency spikes and regional slowdowns sometimes have physical causes. When an entire region degrades, check submarine cable status reports before blaming your stack.
- Multi-region architecture is not paranoia. Cable faults have knocked entire countries onto slow backup routes for weeks. Design for it.
- Invisible maintenance is still maintenance. The subsea industry is what happens when critical work is chronically underfunded because it's out of sight. Every legacy system you maintain deserves better than that fate.
The internet feels like an abstraction because the abstraction usually holds.
It holds because a small number of aging ships and gray-haired splicers keep
quietly closing tickets at the bottom of the ocean. The least we can do is
know they exist — and build our systems as if, one day, a ticket might stay
open a little too long.
Top comments (0)