Originally published on kuryzhev.cloud
A deploy goes out, nothing in the application changed, and yet a slice of users in one region starts seeing timeouts for several minutes before everything quietly recovers. Nobody touched the load balancer. Nobody touched DNS. What actually happened lives one layer down, in BGP routing basics that most application engineers never had to learn until the day a route flap took down their weekend. BGP is not exotic anymore — it runs inside cloud interconnects, between regions, and at the edge of every CDN — so understanding the failure modes is now a normal part of the job, not a networking specialty.
These are independent, practical points. Skip around, verify claims in your own environment, and treat anything with real financial risk as a candidate for a lab test before it hits production.
BGP does not know your traffic is dropping
BGP only cares about reachability, not performance. A path can be fully "up" from BGP's perspective while packet loss on that path is severe. Withdrawal happens when the session goes down, when hold timers expire, when an upstream withdraws the prefix, or when policy changes — not because latency or loss crossed a threshold. This is the single most common source of confusion when engineers assume routing will "route around" a bad path the way an application load balancer would.
If you need loss-aware failover, that has to be built with BFD (Bidirectional Forwarding Detection) or active health checks layered on top of BGP, not BGP alone. Watch out for teams that assume multi-homing alone gives them automatic quality-based failover — it gives redundancy, not intelligence.
Route flapping is usually a symptom, not the disease
When a prefix appears and disappears repeatedly (flapping), the instinct is to blame BGP. In practice the underlying cause is frequently something else: a flaky physical link, an interface resetting, a misconfigured timer, or a route reflector under memory pressure. BGP is often just the messenger reporting instability that already existed at a lower layer.
Route flap dampening exists to suppress the noise, but it has a real cost — accumulated penalties can delay re-advertisement of a prefix after it stabilizes, and how long depends entirely on the configured half-life, reuse, and suppress values. Check dampening state with a command like this on a Cisco IOS-style device:
show ip bgp dampening dampened-paths
# columns show the penalty and the reuse time remaining
# a high penalty on a prefix that "should" be stable is worth investigating
show ip bgp dampening parameters
# confirm the half-life, reuse, suppress and max-suppress values in effectIf dampening is masking a real outage, that can be worse than no dampening at all — it hides the symptom from monitoring for exactly the window when responders need visibility. Note that dampening is disabled by default in many implementations, so confirm whether it is configured before blaming it.
AS_PATH length is not a reliability signal
BGP's best-path selection prefers the shortest AS_PATH, but "shortest" has nothing to do with "healthiest." AS_PATH is also only one step in the decision process — weight, local preference, locally originated routes, and AS_PATH all come before MED and IGP metric, so AS_PATH is decisive only when the earlier tie-breakers are equal. When they are, a three-hop path through a congested transit provider can beat a four-hop path through a well-provisioned one.
For example, a peering session with a regional ISP can look attractive on paper (short AS_PATH) while carrying more loss than a longer path through a better-provisioned backbone. Verify actual path performance with traceroute and packet-loss sampling, not just show ip bgp output, before trusting AS_PATH length as a proxy for quality.
Prefix hijacks are still mostly a config-hygiene problem
Route leaks and hijacks — where a network accidentally or maliciously announces prefixes it doesn't own — remain one of the more consequential BGP failure modes, and they are still largely reducible with basic filtering. RPKI (Resource Public Key Infrastructure) lets a network cryptographically validate that a route announcement comes from an authorized origin AS. Deployment varies considerably by region and operator and is not universal, so treat origin validation as a strong mitigation rather than a guarantee — it does not, on its own, validate the rest of the AS_PATH.
On a router configured with an RPKI validator session, the validation state can be inspected like this:
show ip bgp rpki servers
# confirm the router actually has an established session to a validator
show ip bgp rpki table
# lists prefixes with their registered origin AS and max-length
show ip bgp 203.0.113.0/24
# per-prefix output shows the validation state:
# valid - origin AS matches a ROA
# invalid - origin AS or prefix length conflicts with a ROA
# not found - no ROA covers this prefix; policy decides what happensWatch out for the "not found" state being silently treated the same as "valid" in permissive configurations — that is the gap leaks exploit, and it is a policy decision you have to make explicitly. The validation model itself is specified in RFC 6811 (BGP Prefix Origin Validation) and the broader architecture in RFC 6480; read those before rolling anything out, because the difference between marking invalids and dropping them has real blast radius.
Cloud interconnects hide BGP but don't remove it
Anyone running AWS Direct Connect, Azure ExpressRoute, or Google Cloud Interconnect is running BGP whether they realize it or not — those services establish real BGP sessions with the provider's edge routers. The abstraction is convenient until a session drops and traffic that should have failed over silently doesn't, because a static route or a mismatched BGP community kept the dead path preferred.
For AWS specifically, Direct Connect virtual interfaces expose BGP session state and advertised route counts that are worth checking after any change to on-premises routers; the design considerations are documented in the AWS Direct Connect resiliency guide. A plausible failure here is a BGP session staying "established" while the underlying circuit is degraded — session state alone is not proof the path is healthy, which is precisely why BFD exists on these links.
Communities are how you signal behavior across AS boundaries
BGP communities are often introduced as metadata for filtering or documentation, but they are also a primary lever for requesting routing behavior across AS boundaries — things like "don't advertise this to peers," "prefer this path regionally," or "de-prioritize during maintenance." A missing or incorrect community tag during planned maintenance is a commonly documented cause of unexpected traffic shifts, and it is easy to miss because the configuration itself looks syntactically fine.
The important detail: a community is only a signal. It changes nothing unless the receiving AS has a matching policy, and it is not even transmitted unless the session is configured to send it.
! tag outbound announcements with a community the upstream acts on
ip community-list standard UPSTREAM-DEPREF permit 64512:100
!
route-map SET-COMMUNITY permit 10
set community 64512:100 additive
!
router bgp 64512
neighbor 203.0.113.1 remote-as 64513
neighbor 203.0.113.1 send-community
neighbor 203.0.113.1 route-map SET-COMMUNITY outWithout send-community, the attribute is stripped and the policy silently does nothing. Standard communities also do not carry well-known meaning across providers beyond the reserved ones (NO_EXPORT, NO_ADVERTISE, NO_EXPORT_SUBCONFED), so the numeric values must come from the upstream's published policy. If a provider documents a "graceful maintenance" community, confirm it is actually applied and visible on the receiving side before assuming a maintenance window will be traffic-safe.
Convergence time is the number that actually matters
The metric that predicts user-visible impact during a failure is convergence time — how long it takes every affected router to agree on the new best path after a change. The default timers in the BGP-4 specification (60-second keepalive, 180-second hold time) were designed for stability over speed, though vendor defaults vary and should be confirmed per platform. If a failure is detected by link-down and fast external fallover, teardown is immediate; the painful case is a failure that keeps the interface up — a degraded circuit, a broken path beyond the peer, a black-holing middlebox — where nothing happens until the hold timer expires.
Tuning timers down helps but introduces its own risk: overly aggressive timers cause false-positive session resets under normal jitter. A commonly recommended pattern is BFD for sub-second failure detection paired with less aggressive BGP timers, which separates fast detection from protocol stability, but the right values are vendor- and topology-dependent. The BGP-4 specification (RFC 4271) is the canonical reference for timer semantics when vendor documentation is ambiguous.
None of this replaces reading your specific vendor's or cloud provider's documentation for exact syntax and defaults — behavior varies enough between implementations that assumptions from one platform can quietly break another. For more networking and infrastructure breakdowns written for engineers who'd rather understand the failure than memorize the command, check the rest of the writing at kuryzhev.cloud.
Top comments (0)