On the morning of October 4, 2021, hundreds of millions of people tried to open Facebook, Instagram, and WhatsApp and found nothing. Not an error message. Not a spinning wheel. Nothing. The apps could not connect. The websites could not load. For nearly six hours, one of the most heavily engineered pieces of infrastructure in human history simply ceased to exist from the perspective of the rest of the internet.
The cause was not a cyberattack. It was not a power failure. A software bug did not corrupt a database. A routine configuration change issued by a Facebook engineer, the kind of change that happens at large technology companies dozens of times every day, accidentally told the rest of the internet to stop believing Facebook existed.
The mechanism through which this happened is a protocol called BGP. Most people who use the internet every day have never heard of it. Most developers who build software that runs on the internet have never thought about it. It has no consumer-facing presence, no brand, no product. It is not something you install or configure or monitor in the course of ordinary engineering work. It is infrastructure at the deepest level, the layer beneath the layer beneath the layer that most people ever touch.
It is also, arguably, the most fragile critical system in existence. The entire internet depends on it. It was designed in an afternoon. It has no authentication. It operates on trust. And it breaks regularly.
An afternoon at a cafeteria in Austin
In January 1989, the internet was already in crisis.
The routing technology of the time, a protocol called EGP, the Exterior Gateway Protocol, had been designed for a simpler world. It assumed a hierarchical topology: a core backbone, regional networks hanging off that backbone, smaller networks hanging off those. Traffic flowed up and down the hierarchy according to a rigid tree structure. The design had worked well enough when the internet was a handful of universities and research institutions connected to a government-funded backbone.
By 1989 it was not working at all. The internet had grown too large and too interconnected for a tree. Networks were forming connections that bypassed the hierarchy entirely. Operators were improvising workarounds to make traffic flow correctly, manually managing routing information that EGP could not handle automatically. The system was held together by the expertise and attention of a small number of engineers who understood its internals well enough to intervene when things went wrong.
At the 12th meeting of the Internet Engineering Task Force, held in Austin, Texas, during a lunch break, a software engineer from IBM named Yakov Rekhter sat down with a Cisco engineer named Kirk Lougheed. Between them they had a few napkins and a significant problem to solve.
Over the course of that lunch they sketched a new protocol. The core insight was not complicated. Instead of a hierarchy, treat every network on the internet as an autonomous system: a self-contained unit with its own internal routing policy and its own number, an ASN. Let autonomous systems announce to each other which IP address prefixes they can reach, and include in each announcement the full path of autonomous systems the route has traveled. A network receiving this information knows not just that a route exists, but which networks it passes through to get there.
The path information was the key idea. It solved the routing loop problem that had plagued earlier protocols. If your own autonomous system number appeared in the path of a route someone was advertising to you, you knew that route passed through you to get to them and back, a loop, and you rejected it. No hierarchy needed. No central authority. Each network makes its own decisions about which routes to accept and which to prefer, based on the path information it receives from its peers.
Those napkin sketches became three handwritten pages, which became a draft, which became RFC 1105, the first specification of the Border Gateway Protocol, published in June 1989. The napkins themselves have been lost. Photocopies are preserved at the Cisco headquarters in San Jose.
BGP has been through four major versions since 1989. Version 4, defined in RFC 4271, is what runs the internet today. Its core architecture has not changed in thirty-five years. The protocol designed over lunch to solve the routing crisis of 1989 is still routing every packet that crosses the global internet in 2026.
What BGP actually does
To understand what BGP does, you need to understand what problem it is permanently solving.
The internet is not a single network. It is roughly one hundred thousand separate networks, each owned and operated by a different entity: internet service providers, universities, cloud platforms, content delivery networks, government agencies, large corporations, and thousands of smaller organizations. Each of these networks is an autonomous system.
Your internet provider is an autonomous system. Google is an autonomous system, actually several dozen of them. The network at your university is an autonomous system. Safaricom, Airtel, every carrier operating in East Africa: each one is an autonomous system with its own number.
Inside an autonomous system, routing works however the operator chooses to implement it. OSPF, IS-IS, EIGRP: internal routing protocols that are invisible to the outside world. BGP is what happens at the edges, where one autonomous system meets another.
When your autonomous system wants to reach an IP address that belongs to another autonomous system, BGP is what provides the answer. Every router that participates in global BGP routing maintains a table called the Routing Information Base, a database of IP prefixes and the paths through which they can be reached. These tables are populated by announcements that autonomous systems broadcast to their neighbors: "I can reach 104.16.0.0/12. The path is 15169, 1299, 2914."
That announcement says: to reach the network 104.16.0.0/12, send traffic through autonomous system 15169, then 1299, then 2914. A receiving router adds this information to its routing table, modifies the path to prepend its own ASN, and passes the announcement along to its own neighbors. Within minutes an announcement can propagate to routers around the world.
The global routing table, the complete set of all BGP-announced prefixes visible to a full-feed router anywhere on the internet, currently contains roughly one million prefixes. Every router at the core of the internet holds this entire table in memory and consults it for every forwarding decision. New announcements arrive continuously. Routes are withdrawn when networks go down. The table is never stable, never static, always in the process of being updated by events happening somewhere on the planet.
BGP sessions between neighboring autonomous systems run over TCP port 179. Two routers establish a TCP connection, exchange their routing tables, and then send incremental updates as routes change. If the TCP connection drops, BGP has to re-establish from scratch, which means exchanging the full routing table again and a period during which neither side knows the other's routes. This reconvergence can take minutes, which is an eternity for internet traffic.
The trust problem
BGP has no authentication. This is not an oversight. It is a product of the environment in which it was designed.
In 1989 the internet was a research network. Its operators knew each other. Trust was reasonable. Building authentication into the protocol would have added complexity that seemed unnecessary for a community that already had social trust. The design philosophy of the era was to solve the immediate engineering problem and defer security to later.
Later arrived but the protocol did not change.
When autonomous system A tells autonomous system B that it can reach a particular IP prefix, B has no cryptographic way to verify that A is telling the truth. There is no certificate, no signature, no proof of authorization. A is either lying or it is not. B has no way to know.
This means any autonomous system on the internet can announce any IP prefix, including prefixes it has no right to. The announcement will propagate. Other autonomous systems will accept it. Traffic will flow toward the lying autonomous system, where it might be dropped, inspected, or forwarded to the legitimate destination after being copied.
BGP operates on a principle that networking engineers sometimes call the honor system. You announce what you own. You trust that others are announcing what they own. Most of the time this works because most autonomous systems are operated by organizations that have no reason to lie and significant incentive to maintain good relationships with their peers. The internet runs mostly on mutual dependence and professional reputation.
The problem is that mostly working on a system this large means failures happen constantly.
April 25, 1997
The first major demonstration of how badly BGP can break came on April 25, 1997, from a small network in Florida.
Autonomous system 7007, operated by a company called MAI Network Services and associated with the Florida Internet Exchange, was running a Bay Networks router with an unusual configuration. The router was redistributing its BGP routing table into RIP, an older internal routing protocol, and then redistributing RIP back into BGP. This was not a sensible thing to do, but it was technically possible, and somewhere in that configuration was a catastrophic bug.
RIP did not understand CIDR notation. When the BGP routes passed through RIP and came back out, they had been deaggregated into classful networks, each route broken into smaller pieces. The AS path information had been stripped. Each of these routes was now being announced as if it originated from AS7007.
At around midday, AS7007 started announcing the entire internet routing table back to the internet, all of it, as if AS7007 was the origin of every network on the internet. More than 70,000 networks suddenly had a new, more specific route available to reach their destination: through AS7007.
BGP prefers more specific routes. If there are two routes to a destination, the more specific one wins. The genuine owners of those networks were announcing their prefixes correctly. But AS7007 was announcing smaller, more specific sub-prefixes, and those more specific announcements won.
Traffic from around the world began flowing toward AS7007. The small network in Florida had become, inadvertently, the apparent origin of the entire internet. Its links were nowhere near capable of carrying this load. Traffic was dropped as quickly as it arrived. For network operators watching their routing tables, the effect looked like most of the internet had simultaneously ceased to exist.
The NANOG mailing list, the primary communication channel for network operators at the time, lit up with frantic messages. Engineers at network providers around the world were watching their routing tables fill with AS7007 announcements and scrambling to understand what was happening.
"I happened to be in one of our 7505 routers this afternoon when POP, all of a sudden most of the internet disappeared," one operator wrote to the list. "I immediately thought it was me, but looked around and saw this AS7007 broadcasting MY routes."
Someone called the Florida Internet Exchange. A few moments after the router was disconnected, the internet stabilized. The entire incident lasted roughly two hours.
The post-mortem led to widespread adoption of a practice called prefix filtering: operators would configure their BGP routers to only accept route announcements that made sense given what their peers were supposed to be announcing. If a small regional provider suddenly tried to announce a route for a network belonging to a major carrier on the other side of the world, the filter would catch it.
Prefix filtering helped. It did not solve the problem. It made operators more careful. But careful is not the same as secure, and human attention is not a reliable safeguard for infrastructure this large.
Pakistan Telecom and YouTube, February 2008
In February 2008, the government of Pakistan ordered internet providers in the country to block access to YouTube. The platform had hosted a video deemed anti-Islamic by Pakistani authorities and the government wanted it inaccessible to Pakistani users.
Pakistan Telecom, the state-owned carrier, implemented the block by announcing a more specific BGP route for YouTube's IP prefix. The idea was straightforward: announce a route that covers only YouTube's addresses, pointing traffic toward a null route inside Pakistan's own network. Traffic from Pakistani users would hit the more specific route, get dropped, and YouTube would be effectively blocked within the country. Traffic from users outside Pakistan would continue to use the real YouTube routes and be unaffected.
This is a technically sound approach to blocking. Done correctly, the more specific route stays inside Pakistan's network and is never visible to the rest of the internet.
Pakistan Telecom's upstream provider was PCCW, a major carrier based in Hong Kong. When Pakistan Telecom announced the more specific route for YouTube, it should have been a route visible only within Pakistan's domestic routing. Instead, Pakistan Telecom announced the route to PCCW without the appropriate filters that would have confined it to domestic routing. PCCW accepted the announcement.
PCCW propagated it.
Within two minutes, the more specific route for YouTube's IP space was visible across the global routing table. Traffic heading for YouTube from anywhere on the internet now had a new option: a route that claimed to lead to YouTube but actually led to a null route in Pakistan. The more specific route won.
YouTube disappeared from the internet for approximately two hours. Not just in Pakistan. Everywhere. The entire platform, serving hundreds of millions of users, vanished because a government censorship operation in a single country leaked its routing announcement to an upstream carrier that passed it along without question.
The YouTube network operations team, watching their traffic drop to near zero, identified the problem quickly. They worked with upstream providers to de-prefer the Pakistani routes, and eventually Pakistan Telecom withdrew the announcement. YouTube came back.
The incident was studied extensively in the routing security community. It demonstrated that a single misconfigured BGP announcement, from a single autonomous system, could propagate to routers around the world within minutes and affect traffic globally. The internet's architecture, with its assumption of trust between peers, had no mechanism to prevent this.
The anatomy of a hijack
BGP incidents fall into two categories. A route leak is an accidental propagation of routing information to places it was not supposed to go. A route hijack is a deliberate announcement of a prefix that belongs to someone else.
The distinction matters for attributing intent, but from the perspective of the affected network, the effect is the same: their traffic is going somewhere it should not be.
Route leaks happen constantly. The internet routing table is updated billions of times per day. Operators misconfigure filters, upstream providers fail to apply appropriate validation, and routes end up propagating further than intended. Most of these events are small and self-correcting. The ones that affect major platforms become news.
Route hijacks can be used for several purposes. The most basic is traffic interception: announce a route for your target's IP space, capture the traffic that arrives, and either drop it (a denial of service) or forward it to the real destination after inspecting it (a man-in-the-middle). The legitimate destination may never know the traffic was intercepted.
In 2017, Russian state telecom Rostelecom leaked routing announcements that affected prefixes belonging to major financial institutions. The leak was analyzed extensively and found to include prefixes for Visa, MasterCard, HSBC, and several other financial organizations. Traffic bound for these institutions briefly transited Rostelecom's network. Whether this was accidental or deliberate was never conclusively established.
In 2018, a small internet service provider in Nigeria accidentally leaked routing information that was accepted by Verizon, one of the largest carriers in the United States. The leak caused disruptions for Cloudflare, Google, Facebook, and other major platforms. The Nigerian provider, MainOne, had misconfigured its router. Verizon accepted the announcements without the filters that should have caught them. The combination of two independent failures produced an outage that affected major platforms worldwide.
October 4, 2021
At 15:58 UTC on October 4, 2021, network monitoring systems around the world observed something unusual. Facebook had stopped announcing its BGP routes.
Not modified them. Not announced incorrect ones. Stopped announcing them entirely.
DNS, the system that translates facebook.com into the IP addresses where Facebook's servers live, depends on Facebook's own nameservers. Those nameservers are located at IP addresses that Facebook announces through BGP. Due to Facebook stopping announcing their DNS prefix routes through BGP, every DNS resolver on the internet had no way to connect to their nameservers.
The nameservers themselves were running. Facebook's servers were running. The IP addresses existed and were reachable by people who already had them cached. But the BGP routes that told the rest of the internet where those addresses were had been withdrawn.
From the perspective of every DNS resolver on the internet, Facebook's nameservers had ceased to exist. Queries for facebook.com returned errors. The platforms went dark.
What made this worse than it should have been was a second problem the engineers had not anticipated. After Facebook sent the routing update that caused the outage, its engineers got locked out of the systems that would have allowed them to fix it. The internal network used by Facebook engineers to access infrastructure relied on the same BGP-advertised routes that had just been withdrawn. To change the configuration that had caused the outage, engineers needed access to systems they could no longer reach over the internet.
Engineers had to go physically to data centers, gain access to the hardware, and restore the BGP announcements manually. The access control systems at the data center doors ran on Facebook's internal network. Employees reported being unable to enter buildings because their badges could not be validated against the authentication systems that were offline along with everything else.
It took six hours. Six hours for a platform serving three billion users to come back online because a configuration change withdrew BGP route announcements and the tools needed to fix the problem became unreachable through the same mechanism that caused it.
The security problem nobody has fully solved
Every major BGP incident raises the same question. Why has nothing been done about this? The protocol is nearly forty years old, designed without security, and it breaks the internet with depressing regularity. Why is it still operating on trust?
The honest answer is that fixing BGP is extraordinarily difficult.
The internet is not a single system with a single operator who can push a software update. It is one hundred thousand autonomous systems operated by one hundred thousand different organizations under one hundred thousand different jurisdictions with one hundred thousand different incentive structures. Getting all of them to agree on and implement a security mechanism simultaneously is not a technical problem. It is a coordination problem, which is far harder.
RPKI, the Resource Public Key Infrastructure, is the closest thing to a solution that currently exists. RPKI allows organizations that own IP address space to issue cryptographically signed certificates called Route Origin Authorizations that specify which autonomous systems are permitted to originate which prefixes. A router with RPKI validation enabled can check an incoming BGP announcement against the RPKI database and reject it if the originating AS does not have authorization for that prefix.
As of May 1, 2024, for the first time in the history of RPKI, the majority of IPv4 routes in the global routing table are covered by Route Origin Authorizations. This is genuine progress. For years RPKI coverage was in the single digits. Crossing fifty percent represents a decade of work by routing security advocates pushing individual operators to create authorizations for their address space.
But ROA creation is only half the mechanism. Having a certificate that says you own an address space does not prevent someone from announcing it anyway. The protection only works if the routers receiving the announcement check the certificate and reject the announcement when it fails validation.
Deployment of Route Origin Validation, the enforcement step, lags far behind ROA creation. Networks are reluctant to enable enforcement because rejecting invalid routes can inadvertently block legitimate traffic when ROAs are incorrectly configured. A network operator who enables enforcement and discovers they have broken access to a customer or a peer faces real operational consequences. The incentive to be cautious is strong.
RPKI also addresses only the origin of a route, not the path. Even with full ROA coverage and enforcement, it cannot prevent a legitimate autonomous system from leaking a valid route to a place it should not be. The 2018 Verizon-MainOne incident would not have been stopped by RPKI because the route was originated legitimately. The problem was propagation, not origin.
BGPsec, an extension to BGP that would cryptographically sign the entire AS path, has been specified in RFC standards and studied extensively. It would prevent path manipulation and route leaks in a way that RPKI cannot. It has seen almost no deployment in production because it requires every router in the path to participate, imposes significant computational overhead, and breaks certain common BGP engineering practices. A security mechanism that requires universal adoption before it provides protection is unlikely to achieve that adoption.
The routing table keeps growing
There is a second, slower crisis in BGP that receives less attention than route hijacks but represents an equally serious long-term problem.
The global routing table grows every year. New autonomous systems come online. Existing networks deaggregate their prefixes, splitting larger announcements into smaller ones for more precise routing control. The table has grown from roughly 100,000 prefixes in the mid-2000s to over one million today.
Every router at the core of the internet has to hold this entire table in memory. The memory requirements grow with the table. The processing required to update routing tables when changes arrive grows with the table. The convergence time, the delay between a change in the network and the moment all routers have adjusted their tables to reflect it, grows with the table.
Router vendors have kept pace so far by increasing memory, improving software, and optimizing the data structures used to store and search routing tables. But the growth is not linear. As the internet expands into IPv6 address space, which has a vastly larger address space and the potential for correspondingly larger routing tables, the scaling problem becomes more acute.
The IETF has been studying the routing scalability problem for years. The reports consistently conclude that the current approach cannot scale indefinitely, that a fundamentally different routing architecture will eventually be necessary, and that no consensus exists on what that architecture should look like.
In the meantime, the internet runs on a protocol designed for eighty thousand networks, managing one million prefixes, growing every month.
What this says about the internet
The internet is the most consequential infrastructure project in human history. More economic activity, more communication, more access to knowledge, more coordination of human effort flows through it than through any other system ever built. Its failure, even partial, cascades across every domain that depends on it.
The routing layer that makes it work was designed in an afternoon by two engineers with napkins.
This is not a criticism. The design was brilliant for its time and has proven remarkably durable. The people who built it were solving an urgent problem with the tools and knowledge available in 1989. They could not have anticipated that their solution would scale to one hundred thousand networks and one million routes and three billion users.
What they could not have anticipated is now our inheritance. We live on infrastructure that was built for trust in an environment where trust was reasonable, and we have spent thirty-five years discovering what happens when that trust is violated.
The interesting thing about BGP is not that it breaks. It is that it works as well as it does. The internet, despite everything, routes packets correctly almost all the time. Traffic finds its way from a device in Kisumu to a server in Amsterdam, passing through dozens of autonomous systems operated by organizations that have never met and will never coordinate, following paths announced by routers that accept the word of their peers without verification.
Most of the time, the peers are telling the truth. Most of the time, the announcements are correct. Most of the time, the trust holds.
The incidents that make the news are the exceptions. They are dramatic precisely because the baseline is so reliable. A protocol designed in an afternoon, running the global internet for thirty-five years, breaking only occasionally despite having no authentication, no verification, no enforcement, and no central authority: this is either a testament to the power of good design under constraints, or a warning about how long we can depend on trust as a security model.
Probably both.
Top comments (0)