DEV Community

Ripan Deuri
Ripan Deuri

Posted on AI-assisted

PCI Enumeration Explained

#architecture #hardware #systems

Introduction

PCIe enumeration is the process by which the operating system discovers all devices in the system, assigns them unique identifiers (BDFs), and configures their resources.

This article walks through PCIe enumeration using a simple topology and shows how configuration transactions (CfgRd/CfgWr) and completion packets (CplD) are used during device discovery and setup.

Topology 

                +----------------------+
                |   SoC (Root Complex) |
                +----------+-----------+
                           | Bus 0
        -----------------------------------------
        |                                       |
   +---------+                          +---------------+
   |  EP1    |                          |     Switch    |
   +---------+                          +-------+-------+
                                                | Bus 1
                                           +---------+ 
                                           |  EP2    |
                                           +---------+
Enter fullscreen mode Exit fullscreen mode

This topology represents a typical PCIe hierarchy:

  • The Root Complex (RC) initiates all configuration transactions
  • Bus 0 is directly connected to the RC
  • EP1 is an endpoint on Bus 0
  • The switch is also on Bus 0 and creates a secondary bus
  • EP2 is located behind the switch on Bus 1

ECAM base address: 0x20000000

Initial BDF assignments:

  • RC: Bus 0, Device 0, Function 0
  • EP1: Bus 0, Device 1, Function 0
  • Switch: Bus 0, Device 2, Function 0

Phase 1 - Scanning Bus 0

The enumeration process begins with the RC scanning Bus 0. It probes each device number by issuing configuration read requests to offset 0x00 (Vendor ID / Device ID).

A valid completion indicates the presence of a device.

Probe EP1 (Bus 0, Device 1, Function 0)

CfgRd0 TLP (RC -> EP1) 

--------------------------------------------------------
DW0
Byte 0:   0x04    Fmt=000, Type=00100 (CfgRd0)
Byte 1:   0x00    TC=0, Attr=0
Byte 2:   0x00
Byte 3:   0x01    Length = 1 DW

DW1
Byte 4:   0x00    Requester ID [15:8] (Bus=0)
Byte 5:   0x00    Requester ID [7:0]  (Dev=0, Fn=0)
Byte 6:   0x01    Tag = 1
Byte 7:   0x0F    LastBE=0000, FirstBE=1111

DW2
Byte 8:   0x00    Bus Number = 0
Byte 9:   0x08    Device=1, Function=0
Byte 10:  0x00    Register Number [15:8]
Byte 11:  0x00    Register Number [7:2], Rsvd
--------------------------------------------------------
Enter fullscreen mode Exit fullscreen mode

Completion TLP (EP1 -> RC) 

--------------------------------------------------------
DW0
Byte 0:   0x4A    Fmt=010, Type=01010 (CplD)
Byte 1:   0x00    TC=0
Byte 2:   0x00
Byte 3:   0x01    Length = 1 DW

DW1
Byte 4:   0x00    Completer ID [15:8] (Bus=0)
Byte 5:   0x08    Completer ID [7:0]  (Dev=1, Fn=0)
Byte 6:   0x00    Status=000 (Successful)
Byte 7:   0x04    Byte Count = 4 bytes

DW2
Byte 8:   0x00    Requester ID [15:8]
Byte 9:   0x00    Requester ID [7:0]
Byte 10:  0x01    Tag = 1
Byte 11:  0x00    Lower Address

DW3 (Data)
Byte 12:  0x8C
Byte 13:  0x16
Byte 14:  0x09
Byte 15:  0x11
--------------------------------------------------------
Enter fullscreen mode Exit fullscreen mode

The device responds with a valid Vendor ID and Device ID, confirming its presence.

Assigned BDF: 0000:00:01.0

Probe Switch (Bus 0, Device 2, Function 0)

The same process is repeated for the switch.

  • A valid completion confirms the device
  • The Header Type register (offset 0x0E) is read
  • Value indicates a PCIe-to-PCIe bridge (Type 1 header)

Assigned BDF: 0000:00:02.0

Probe Remaining Devices

Devices from 3 to 31 are probed. No responses are received.

Bus 0 scan is complete.

Phase 2 - Configuring the Switch

Once the switch is identified as a bridge, the RC assigns bus numbers to enable access to downstream devices.

Bus Number Assignment

For Bus 0, Device 2, Function 0:

  • Primary Bus Number = 0
  • Secondary Bus Number = 1
  • Subordinate Bus Number = 0xFF (temporary upper limit)

Configuration Write 

CfgWr0: Bus=0, Dev=2, Fn=0, Reg=0x018
Data: 0x00FF0100
Enter fullscreen mode Exit fullscreen mode

Routing Behavior of the Switch

After configuration:

  • If Bus Number equals Secondary Bus -> forward downstream
  • If Bus Number is greater than Secondary -> forward further downstream
  • Otherwise -> do not forward

This routing logic is used for Type 1 configuration transactions.

Phase 3 - Scanning Bus 1

The RC now scans Bus 1 using Type 1 configuration transactions.

Probe EP2 (Bus 1, Device 0, Function 0)

CfgRd1 TLP (RC -> Switch) 

--------------------------------------------------------
DW0
Byte 0:   0x05    Fmt=000, Type=00101 (CfgRd1)
Byte 1:   0x00
Byte 2:   0x00
Byte 3:   0x01

DW1
Byte 4:   0x00    Requester ID (Bus=0, Dev=0, Fn=0)
Byte 6:   0x03    Tag = 3

DW2
Byte 8:   0x01    Bus Number = 1
Byte 9:   0x00    Device=0, Function=0
--------------------------------------------------------
Enter fullscreen mode Exit fullscreen mode

Switch Processing

  • Bus Number matches Secondary Bus (1)
  • Switch converts CfgRd1 to CfgRd0
  • Forwards request to EP2

Completion Flow

EP2 -> Switch 

Completion TLP (CplD)
Completer ID = Bus 1, Dev 0, Fn 0
Requester ID = Bus 0, Dev 0, Fn 0
Tag = 3
Enter fullscreen mode Exit fullscreen mode

Switch -> RC

  • Switch forwards completion upstream
  • Packet remains unchanged

Assigned BDF: 0000:01:00.0

Key Observation

  • Requests are routed using Bus Number
  • Completions are routed using Requester ID

This distinction is essential for understanding PCIe routing behavior.

Phase 4 - Finalizing Bus Numbers

After scanning Bus 1:

  • No additional bridges are found
  • Highest bus number is 1

Update subordinate bus number:

CfgWr0: Bus=0, Dev=2, Fn=0, Reg=0x018
Data: 0x00010100
Enter fullscreen mode Exit fullscreen mode

BAR Assignment

After enumeration, the kernel assigns memory regions to devices.

EP1 BAR Assignment 

CfgWr0: Bus=0, Dev=1, Fn=0, Reg=0x010, Data=0xFFFFFFFF
CfgRd0: Bus=0, Dev=1, Fn=0, Reg=0x010
CfgWr0: Bus=0, Dev=1, Fn=0, Data=0x30000000
Enter fullscreen mode Exit fullscreen mode

EP2 BAR Assignment

Note: EP2 is at Bus=1, Device=0, Function=0

CfgWr1 -> Switch -> CfgWr0: Reg=0x010, Data=0xFFFFFFFF
CfgRd1 -> Switch -> CfgRd0: Read size
CfgWr1 -> Switch -> CfgWr0: Data=0x30200000
Enter fullscreen mode Exit fullscreen mode

Switch Memory Window 

CfgWr0: Bus=0, Dev=2, Fn=0, Reg=0x020 = 0x30200000
CfgWr0: Bus=0, Dev=2, Fn=0, Reg=0x024 = 0x303FFFFF
Enter fullscreen mode Exit fullscreen mode

This allows the switch to forward memory transactions to EP2.

Final Topology 

RC (Bus 0)
|
|---- EP1: 0000:00:01.0  BAR0=0x30000000
|
|---- Switch: 0000:00:02.0
              Primary=0 Secondary=1 Subordinate=1
              |
              |---- EP2: 0000:01:00.0  BAR0=0x30200000
Enter fullscreen mode Exit fullscreen mode

Conclusion

PCIe enumeration follows a structured and predictable sequence:

  • Devices are discovered using configuration reads
  • Bridges are configured with bus numbers
  • Downstream buses are scanned using Type 1 transactions
  • Routing is based on Bus Number (requests) and Requester ID (completions)
  • Resources such as BARs are assigned after discovery

Top comments (0)