DEV Community

Ashwin
Ashwin

Posted on

How to debug why your PCIe device doesn't enumerate during bring up

#pci #linux

Notes from bringing up a PCIe WiFi module on i.MX8MQ; symptoms and how to diagnose them.

Phy link never came up — what does this mean?

This message is typically seen in dmesg as shown below.

[    3.828121] imx6q-pcie 33800000.pcie: iATU: unroll T, 4 ob, 4 ib, align 64K, limit 4G
[    4.807241] imx6q-pcie 33c00000.pcie: Phy link never came up
[    4.841482] imx6q-pcie 33800000.pcie: Phy link never came up
[    5.821279] imx6q-pcie 33c00000.pcie: Phy link never came up
[    5.830481] imx6q-pcie 33c00000.pcie: PCI host bridge to bus 0001:00
[    5.854997] imx6q-pcie 33800000.pcie: Phy link never came up
[    5.862205] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
Enter fullscreen mode Exit fullscreen mode

It means one of the following

  1. The PCIe peripheral is not powered up.
  2. PCIe reset is not deasserted, so the chip is in reset. This could be because the DTB is deasserting an incorrect GPIO.
  3. Reference clock is not enabled
  4. Using the incorrect PCIe controller in the device tree.
  5. As we can see that both the PCIe controllers can report this. So first determine which controller is the peripheral hooked to. More on this in the next section.

Which PCIe controller is my device on? (&pcie0 vs &pcie1)

The rule here is to match by address and not by label/name. If the schematic calls out controllers as PCIE1 and PCIE2, and the device tree lists pcie0 and pcie1, understand the mapping. The DTS label is arbitrary - match by register base (@address in the node name), which is the same in the DTS reg and the reference memory map. For definitive addresses look in the .dtsi, as sometimes the manuals are misleading.

Given below is a mapping table for i.MX8MQ

DTS.     |  ADDRESS (in .dtsi)  | Silicon (RM) Label
&pcie0   | 0x33800000          |     PCIe1 
&pcie1   | 0x33c00000          |     PCIe2
Enter fullscreen mode Exit fullscreen mode

The addresses are listed in the chip’s memory layout are from processor reference manual. Below is snapshot from the i.MX8MQ reference manual, where the layout for the core A-53 is listed.

Start Address | End Address |  Size   | Description
3381_0000     | 3381_3FFF   |  4MB    | PCIe-2   << incorrect in the RM
3380_0000     | 3380_3FFF   |  4MB    | PCIe-1
Enter fullscreen mode Exit fullscreen mode

As you can see from above there are two discrepancies:

  1. The size is listed at 4MB (0x400000), but the end address points to a 16K address space.
  2. PCIe-2's address region lies within PCIe-1, which is incorrect. Lesson: When in doubt, look at the definitions in .dtsi 
pcie0: pcie@33800000 {
pcie1: pcie@33c00000 {
Enter fullscreen mode Exit fullscreen mode

So we know from the schematic, the peripheral is on PCIe-1, which maps to &pcie0 in the DTB. So focus on &pcie0. As a next step validate that the reset-gpio of the selected peripheral matches to whats is specified in the device tree. If it does not match, change the GPIO specified in the device tree to match.

lspci shows my device but a Kernel driver in use is not seen in the output

This could be the next possible step, where lspci reports the device is enumerated but the bars are disabled. If the bars are enabled -> Hurrah!! You are done. The remainder of this section is for instances where the output of lspci looks as so:

0000:01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4356 802.11ac Wireless Network Adapter (rev 02)
        Subsystem: Broadcom Inc. and subsidiaries BCM4356 802.11ac Wireless Network Adapter
        Flags: fast devsel, IRQ 222
        Memory at 18400000 (64-bit, non-prefetchable) [disabled] [size=32K] << BAR not setup
        Memory at 18000000 (64-bit, non-prefetchable) [disabled] [size=4M] << BAR not setup
        Capabilities: [48] Power Management version 3
Enter fullscreen mode Exit fullscreen mode

and no Kernel driver in use shows up in the output. This means that the driver has not claimed the device and setup the BARs. In summary, the PCIe core enumerated the device but driver is not binding.

For such cases, do the following steps. Note that the steps are written for a WiFi PCIe card on the i.MX8MQ eval kit; meant to serve as an example. Adapt it for your specific bring up.

  1. Check what the bus, the driver is built for 
zcat /proc/config.gz | grep -i brcmfmac
Enter fullscreen mode Exit fullscreen mode
  1. Check if the driver is built for the wrong bus, example SDIO enabled but not PCIE 
CONFIG_BRCMFMAC=m
CONFIG_BRCMFMAC_SDIO=y   << Will not work for PCIe
# CONFIG_BRCMFMAC_PCIE is not set   << Should be there for PCIe
Enter fullscreen mode Exit fullscreen mode

In the example above, the driver exists but ignores the PCIe device.

  1. Enable PCIE with CONFIG_BRCMFMAC_PCIE=y, optionally remove SDIO, rebuild, and load on the device. It might be a good idea to check your new kernel config has the option set, before loading it on device. 
CONFIG_BRCMFMAC=m
CONFIG_BRCMFMAC_SDIO=n
CONFIG_BRCMFMAC_PCIE=y   << THIS
Enter fullscreen mode Exit fullscreen mode

Once loaded on the device, ensure that the driver is loaded.

lsmod | grep -i brcmfmac
brcmfmac_wcc           12288  0
brcmfmac              233472  1 brcmfmac_wcc
brcmutil               16384  1 brcmfmac
Enter fullscreen mode Exit fullscreen mode

You will need to determine if the driver firmware is available to load. Check /usr/lib/firmware to see if it’s available.
In case you are wondering where did the BAR addresses come from, these come from the PCIe memory regions defined in the chip address map. For example for the Cortex A-53:

Start Address  | End Address | Region  | Size   | Description
2000_0000      | 27FF_FFFF   | PCIe-2  | 128MB  | PCIe-2
1800_0000      | 1FFF_FFFF   | PCIe-1  | 128MB  | PCIe-1
Enter fullscreen mode Exit fullscreen mode

Here we can see that the chip lies in the PCIe-1 region of the silicon -> &pcie0 of device tree, as we verified in the previous section.

Run lspci -v for a final confirmation.

lspci -v
snip … snip
0000:01:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4356 802.11ac Wireless Network Adapter (rev 02)
        Subsystem: Broadcom Inc. and subsidiaries BCM4356 802.11ac Wireless Network Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 234
        Memory at 18400000 (64-bit, non-prefetchable) [size=32K] << BAR is setup
        Memory at 18000000 (64-bit, non-prefetchable) [size=4M] << BAR is setup
        Capabilities: [48] Power Management version 3
        Capabilities: [58] MSI: Enable+ Count=1/16 Maskable- 64bit+
        Capabilities: [68] Vendor Specific Information: Len=44 <?>
        Capabilities: [ac] Express Endpoint, IntMsgNum 0
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Device Serial Number 00-00-38-ff-ff-00-38-0f
        Capabilities: [150] Power Budgeting <?>
        Capabilities: [160] Virtual Channel
        Capabilities: [1b0] Latency Tolerance Reporting
        Capabilities: [220] Physical Resizable BAR
        Capabilities: [240] L1 PM Substates
        Kernel driver in use: brcmfmac << THIS too!!
        Kernel modules: brcmfmac
Enter fullscreen mode Exit fullscreen mode

Additional points

  1. The 100 Mhz reference clock is either provided internally by the i.MX8M or an external clock generator IC on the board. If an external clock generator is used, validate that the chip is enabled.
  2. Check that the reset polarity specified in the device tree is correct. In the case of PCIe, it is an active low.
  3. Verify that the fixed-voltage regulators regulator-fixed for PCIe are enabled in the device tree. These supply a constant 3.3v to the PCI slot.
  4. Specifically for Broadcom drivers, these need a .txt NVRAM configuration file, and a .bin file present in /usr/lib/firmware directory, otherwise they fail silently. It will manifest as wlan0 not appearing. In such cases run dmesg | grep brcm to triage.

Top comments (0)