pathvector-dev

Posted on Aug 1 • Originally published at blog.pathvector.dev

Announce, withdraw, and peer down are three different code paths

#python #network #bgp #protocol

Originally published at https://blog.pathvector.dev/protocol-in-code-bgp-13/ — part of the free Protocol Lab series.

This post is part of Protocol in Code, a free series that reads network protocols as logic — inputs, state, and branches — rather than as configuration examples. Every module points at a real Python file you can open, read, and run; the source lives at github.com/pathvector-studio/protocol-in-code. If you're newer to this and want to build up hands-on first, start with the companion Protocol Lab series, which is the practical, run-it-yourself track.

The question

When a BGP speaker receives news from the outside world, that news arrives in a few distinct flavors. A peer announces a prefix. A peer withdraws a prefix. A peer's session drops entirely. It's tempting to model all three as "an update happened, go recompute" — one generic entry point with a flag or two.

So here's the question to hold onto while you read:

Core question: What function decides which control-plane path to run for announce, withdraw, and peer-down events?

And its sharper follow-up: why are these three separate functions instead of one? The answer is not stylistic. Each event enters the control plane at a different point, and one of them can touch an unbounded number of prefixes.

The file is src/protocol_in_code/bgp/events.py.

Three triggers, not one

Start at the top of the file. The events are declared as three separate frozen dataclasses, and the shape of each one already tells you something:

@dataclass(frozen=True)
class AnnounceEvent:
    peer_id: str
    prefix: str
    attributes: PathAttributes


@dataclass(frozen=True)
class WithdrawEvent:
    peer_id: str
    prefix: str


@dataclass(frozen=True)
class PeerDownEvent:
    peer_id: str

Read them as a narrowing sequence. An announce carries the full payload: who, what prefix, and what path attributes came with it. A withdraw carries who and what prefix — there are no attributes, because you're removing a path, not describing one. A peer-down carries only who. It doesn't name a prefix at all, and that omission is the whole reason the third branch has to be written differently: the set of affected prefixes isn't in the event, it has to be discovered from state.

All three converge on one return type:

@dataclass(frozen=True)
class EventResult:
    accepted: bool
    touched_prefixes: tuple[str, ...]
    best_paths: dict[str, PathCandidate | None]
    export_changes: tuple[ExportChange, ...]

Note touched_prefixes is a tuple and best_paths is a dict keyed by prefix. The return type was designed for the many-prefix case even though two of the three branches will only ever fill it with one entry.

The shape of the dispatch

Before walking through the real code, here's the shape in pseudocode. This is the lesson; the actual functions are longer, but they don't deviate from it:

if announce:
    gate peer
    recompute prefix
    refresh exports
elif withdraw:
    remove received path
    recompute prefix
    refresh exports
elif peer_down:
    remove whole peer table
    recompute affected prefixes
    refresh exports

Three things stand out. Only the announce branch has a gate. Every branch ends in the same two steps — recompute, then refresh exports. And only the last branch has a loop.

Announce: gate first, then recompute

process_announce_event() starts with the session gate that Session 11 made explicit:

    peer = peers[event.peer_id]
    accepted = receive_update_if_established(adj_rib_in, peer, event.prefix, event.attributes)
    if not accepted:
        return EventResult(
            accepted=False,
            touched_prefixes=(event.prefix,),
            best_paths={event.prefix: loc_rib.best_paths.get(event.prefix)},
            export_changes=(),
        )

An announce from a peer that isn't in ESTABLISHED doesn't get to write into the Adj-RIB-In at all. Look carefully at the early return: it doesn't just bail with a flag. It reports touched_prefixes containing the prefix, and it reports the current best path from loc_rib.best_paths.get(event.prefix) — the one that was already there. export_changes is empty. The rejected announce is visible in the result, but it changed nothing.

That's a deliberate distinction worth internalizing: "we looked at this prefix" and "this prefix changed" are different claims, and the result type keeps them separate.

If the gate passes, the rest is two calls:

    best = _recompute_prefix(adj_rib_in, loc_rib, event.prefix, vrps, policies)
    changes = refresh_exports_for_prefix(event.prefix, loc_rib, adj_rib_out, export_targets)

_recompute_prefix() is where the event layer hands off all the prefix-level decision work:

def _recompute_prefix(
    adj_rib_in: AdjRIBIn,
    loc_rib: LocRIB,
    prefix: str,
    vrps: list[VRP],
    policies: PipelinePolicies,
) -> PathCandidate | None:
    best = select_best_installable_for_prefix(adj_rib_in, prefix, vrps, policies)
    if best is None:
        remove_best_path(loc_rib, prefix)
        return None

    install_best_path(loc_rib, best)
    return best

This helper is the reason the three branches stay short. Every hard question — which candidate wins the decision process, whether RPKI validation rejects it, what the policies say — lives behind select_best_installable_for_prefix(). The event layer only cares about the binary outcome: there's a best path (install it) or there isn't (remove whatever was installed). Session 14 goes deeper into that decision logic, but the dispatch shape you're reading here doesn't change.

Withdraw: no gate, and it starts by deleting

Now compare process_withdraw_event():

def process_withdraw_event(
    event: WithdrawEvent,
    adj_rib_in: AdjRIBIn,
    loc_rib: LocRIB,
    adj_rib_out: AdjRIBOut,
    export_targets: tuple[ExportTarget, ...],
    vrps: list[VRP],
    policies: PipelinePolicies,
) -> EventResult:
    withdraw_received_path(adj_rib_in, event.peer_id, event.prefix)
    best = _recompute_prefix(adj_rib_in, loc_rib, event.prefix, vrps, policies)
    changes = refresh_exports_for_prefix(event.prefix, loc_rib, adj_rib_out, export_targets)

Two differences from announce, and both are in the signature and the first line.

First, look at the parameter list: there's no peers argument. The withdraw path doesn't have access to the session table, which means it structurally cannot gate on session state. That's not an omission — it's the design being enforced by the type signature.

Second, the first statement is a removal, not a validation. There is no accepted check and the function always returns accepted=True. Withdrawing state you may or may not have is safe; adding state you shouldn't have is not. Removal is idempotent in a way that installation isn't, so the asymmetry is justified rather than sloppy.

After that first line, the two branches are identical: recompute, refresh. The withdrawal might not even change the best path — if the withdrawing peer wasn't the winner, select_best_installable_for_prefix() will return the same candidate it did before, and refresh_exports_for_prefix() will produce no changes. The code doesn't special-case that. It recomputes unconditionally and lets the export refresh decide whether anything is actually different.

Peer down: one event, many prefixes

process_peer_down_event() is where the shape breaks:

    lost_prefixes = tuple(adj_rib_in.paths_by_peer.get(event.peer_id, {}).keys())
    adj_rib_in.paths_by_peer.pop(event.peer_id, None)
    peer = peers.get(event.peer_id)
    if peer is not None:
        peers[event.peer_id] = PeerSession(peer_id=peer.peer_id, config=peer.config, state=SessionState.ACTIVE)

The first line is the crux of this whole module. Because PeerDownEvent carries no prefix, the affected set has to be read out of state before that state is destroyed. Order matters: snapshot the keys, then pop the peer's table. Reverse those two lines and you lose the list of prefixes you were about to fix up.

Note also .get(event.peer_id, {}) and .pop(event.peer_id, None) — a peer-down for a peer with no received paths, or no entry at all, is a no-op rather than a KeyError. Session teardown races are exactly the situation where you get duplicate or spurious down events.

The session state moves to ACTIVE, not IDLE. That's the FSM's "trying to connect" state, which is where a peer that just dropped should sit. And because PeerSession is frozen, the update is a replacement of the dict entry, not a mutation.

Then comes the loop that no other branch has:

    lost: dict[str, PathCandidate | None] = {}
    changes: list[ExportChange] = []
    for prefix in lost_prefixes:
        lost[prefix] = _recompute_prefix(adj_rib_in, loc_rib, prefix, vrps, policies)
        changes.extend(refresh_exports_for_prefix(prefix, loc_rib, adj_rib_out, export_targets))

    return EventResult(
        accepted=True,
        touched_prefixes=tuple(lost.keys()),
        best_paths=lost,
        export_changes=tuple(changes),
    )

This is the same two-step body as announce and withdraw — _recompute_prefix() then refresh_exports_for_prefix() — wrapped in a for. And note what it does not do: it doesn't blanket-withdraw every prefix the peer had announced. For each lost prefix it re-runs the decision process against what's left in the Adj-RIB-In. If another peer was also announcing that prefix, _recompute_prefix() finds it and installs it, and the export refresh emits a change to the new next hop rather than a withdrawal. Only prefixes where the downed peer was the sole source collapse to None.

That's the practical answer to the "which branch touches many prefixes" question, and it's why a single generic update function would have been awkward: two of the three branches know their prefix up front, and one has to go find it.

Note: Recomputing per prefix in a loop is also where real BGP implementations spend serious engineering effort. A peer down on a full-table session means several hundred thousand recomputations. Real implementations batch, defer, and prioritize; this toy model just loops.

Reading lens: the same shape shows up elsewhere

The pattern here — the event doesn't name its own blast radius, so you snapshot state before destroying it — isn't BGP-specific. It's the same shape as conntrack entry expiry, where a single timeout has to enumerate the flows it invalidates before dropping the table entry. It's the same shape as a DNS resolver flushing everything under a zone when the zone's delegation changes. In each case the trigger is small and singular, and the cleanup has to derive its own scope from state that's about to go away.

Once you've seen the announce/withdraw/peer-down split, you'll notice that most protocol event loops have exactly this trio: a gated additive event, an ungated removal, and a teardown that fans out.

Run it

The walkthrough for this session is runnable:

PYTHONPATH=src python3 examples/bgp/session_13_walkthrough.py

Watch for four things in the output, in order:

one announce producing a best path
one withdraw removing that best path
a second announce restoring it
one peer-down withdrawing all affected exports

The last one is the payoff. Compare the length of touched_prefixes in that final result against the three before it.

Toy model boundary

This is a teaching model, and it simplifies in ways that matter if you carry the mental model into production:

Events are synchronous and serialized. Each process_*_event() call runs to completion before the next event exists. Real BGP speakers process a stream of UPDATE messages off a TCP socket, with input queues, read batching, and no guarantee that a peer-down is observed before or after the updates that were in flight when the session dropped.
A real UPDATE message carries both announcements and withdrawals. RFC 4271's UPDATE has a Withdrawn Routes field and an NLRI field in the same message. Splitting them into AnnounceEvent and WithdrawEvent is a modeling choice that makes the branches legible; a real parser has to handle both in one message and get the ordering right.
No timers. There's no MRAI (Minimum Route Advertisement Interval), no route flap damping, no graceful restart. In this model, peer-down means the paths are gone immediately. Real BGP with graceful restart keeps stale routes marked and installed while it waits for the session to come back.
The recompute is exhaustive, not incremental. _recompute_prefix() re-runs the full selection for a prefix every time. Production implementations track which candidate is currently best and short-circuit most updates without a full re-scan.
export_changes is a return value, not I/O. Nothing is transmitted. There are no OPEN/KEEPALIVE/NOTIFICATION messages, no hold timer, no actual peers.
peers[event.peer_id] in the announce path will KeyError on an unknown peer. That's fine for a model where you control the inputs; a real speaker never trusts the peer identifier that far.

Check yourself

Try to answer these by reading events.py alone, without running anything. If you can't, that's the signal for where to look:

Why is process_withdraw_event() missing the peers parameter that the other two branches take, and what would change if you added a session gate to it?
In process_peer_down_event(), what breaks if you swap the order of the lost_prefixes assignment and the paths_by_peer.pop() call?
An announce arrives from a peer in OPEN_CONFIRM. Trace the return value: what is accepted, what is in touched_prefixes, and what is in best_paths?

Once you have answers, go verify them against the source rather than against your memory of this post.

Done when

You can explain why announce, withdraw, and peer down are three branches instead of one generic update function — and you can say, without looking, which branch touches exactly one prefix and which one can touch many.

DEV Community