A network engineer brings up a new eBGP peering session with a smaller regional ISP customer, mostly for redundancy on one edge router. The session comes up clean, prefixes flow, ping tests look fine, the change gets marked done. Three weeks later, users start complaining that a specific set of external sites are slow or occasionally unreachable, but only some sites, only sometimes, and every other destination on the internet is fine. Nothing on the core network changed. The firewall team clears itself first, then the DNS team, then the ISP's own status page, all clean. It takes three days to find, because the failure doesn't look like a network problem, it looks like a handful of unrelated websites having a bad week.
The actual cause: that regional ISP customer had its own upstream provider, and somewhere in its own network it started re-advertising prefixes it had learned from that upstream back out over the peering session, instead of only announcing the small block of address space it was actually assigned. BGP doesn't have an opinion about whether a route announcement is honest, it just evaluates whatever gets advertised on its own merits, best AS-path length, local preference, and so on, and picks a best path from what it's told. For a specific subset of external prefixes, the path through the small regional customer looked shorter or got preferred over the path through the real, well-provisioned upstream. Traffic to those prefixes started routing out over a link built to carry that customer's own modest traffic, not a chunk of everyone else's internet-bound traffic too. That link congested exactly the way an undersized pipe does under load it was never sized for: intermittently, for specific destinations, at specific times of day, which is precisely the pattern that gets mistaken for a dozen unrelated small problems instead of one root cause.
There was never any inbound filtering on that eBGP session restricting what the customer was allowed to announce. No prefix-list matching only their assigned block, no AS-path filter rejecting anything that wasn't originated by their own AS, no max-prefix limit that would have flagged an unexpected jump in announced routes and torn the session down before it did damage. The session was configured to accept whatever the neighbor sent, which is the default behavior of eBGP and exactly the behavior RFC 7454 (BGP Operations and Security) tells operators not to leave in place: every eBGP session, especially one to a customer or peer rather than a transit provider you fully trust, should have an explicit inbound policy limiting announcements to what that neighbor is actually supposed to originate.
Finding it, once someone thought to look, was a matter of checking the routing table for the affected prefixes and seeing which neighbor and AS-path they were actually arriving through, then walking backward to notice it wasn't the expected upstream at all. The fix was two commands: a prefix-list scoped to the customer's actual assigned block, applied inbound, plus a max-prefix limit as a safety net for the next time. Nothing about the fix was hard. The three days went into realizing a routing policy gap, not a server or an application, was the actual cause of a symptom that looked like neither.
This is exactly the kind of senior-level BGP hygiene the Network Engineer L3 book covers across 307 pages, 25 labs and 10 case studies: not just how to bring a BGP session up, but the operational discipline (route filtering, max-prefix limits, path selection under real multihoming) that keeps one misbehaving neighbor from quietly reshaping your traffic: https://resources.codelivly.com/product/network-engineer-l3/
Top comments (0)