DEV Community

Ripan Deuri
Ripan Deuri

Posted on

Understanding PCIe Data Link Layer

#architecture #computerscience #networking

1. Introduction

PCI Express (PCIe) uses a layered architecture to separate concerns like transaction creation, reliability, and physical transmission. The Data Link Layer (DLL) ensures reliable communication between directly connected devices.

The Transaction Layer generates packets (TLPs), and the Physical Layer transmits bits. The Data Link Layer sits between them, guaranteeing that TLPs are delivered correctly, in order, and without corruption over a single PCIe link.

2. PCIe Layers

  • Transaction Layer (TL): Creates Transaction Layer Packets (TLPs)
  • Data Link Layer (DLL): Ensures reliable delivery of TLPs
  • Physical Layer (PHY): Handles signaling and bit transmission

3. Data Link Layer (DLL)

3.1 What Data Link Layer Does

Responsibilities of DLL are:

  • Reliable delivery: Sequence numbers + ACK/NAK + Replay buffer
  • Error detection: LCRC (Link CRC) on every TLP
  • Flow control: Credit based system to prevent buffer overflow

DLL operates on two types of packets:

  • TLPs (Transmission Layer Packets) - actual data from the Transmission Layer
  • DLLPs (Data Link Layer Packets) - control packets used by the DLL itself (ACK, NAK, flow control updates)

3.2 Sequence Numbers

Each outgoing TLP is assigned a sequence number and LCRC by the Data Link Layer.

+--------------------------------------+
| Seq#   |    TLP Payload    | LCRC    |
| 12-bit |                   | 32-bit  |
+--------------------------------------+
Enter fullscreen mode Exit fullscreen mode

DLL Tx steps:

  • Assign next sequence number
  • Prepend seq#
  • Compute LCRC over [Seq# | TLP]
  • Append LCRC
  • Save copy in Replay buffer
  • Send [Seq# | TLP | LCRC] to Physical Layer

DLL Rx steps:

  • Receive [Seq# | TLP | LCRC] from Physical Layer
  • Recompute LCRC - does it match?
    • YES: continue
    • NO: send NAK DLLP, discard TLP
  • Check Seq# - is it the expected one?
    • YES: continue
    • NO: send NAK DLLP, discard TLP
  • Strip Seq# and LCRC
  • Pass TLP up to Transaction Layer
  • Send ACK DLLP back to the transmitter

3.3 ACK/NAK Protocol

Replay Buffer
The transmitter keeps a Replay Buffer - a copy of every TLP that has been sent but not yet acknowledged.

Replay Buffer:
+--------------------------------+
| Seq = 40 | Seq = 41 | Seq = 42 |
+--------------------------------+
      ^
      |
     Once ACK(40) is received, Seq = 40 is removed
Enter fullscreen mode Exit fullscreen mode

ACK
When the receiver successfully receives a TLP, it sends back an ACK DLLP with the sequence number of the last successfully received TLP.

ACK is cumulative. An ACK (Seq = 41) means TLPs are received TLPs upto and including Seq = 41.

NAK
When the receiver detected a corrupted TLP (bad LCRC or an out of order sequence number), it sends a NACK DLLP.

Upon receiving a NACK, the transmitter replays all unACK'd TLPs from the replay buffer, starting from the NAK'd sequence number.

A Complete ACK/NAK Example

RC DLL                                  EP DLL
  |                                       |
  |-- TLP (Seq=41) ---------------------->| OK
  |-- TLP (Seq=42) ---------------------->| OK
  |-- TLP (Seq=43) ---------------------->| CRC Error!
  |                                       |
  |<-- ACK (Seq=42)-----------------------| ACKs Seq=41 and Seq=42 cumulatively
  |<-- NAK (Seq=43)-----------------------| NAK Seq=43, requests retransmit
  |                                       |
  | [Transmitter replays from Seq = 43]   |
  |-- TLP (Seq=43)----------------------->| OK
  |-- TLP (Seq=44)----------------------->| OK
  |                                       |
  |<-- ACK (Seq=44)-----------------------| ACKs Seq=43 and Seq=44 cumulatively
Enter fullscreen mode Exit fullscreen mode

If the transmitter does not receive an ACK within a timeout period, it assumes the TLP was lost and replays all unACK'd TLPs. This handles the case where the ACK itself was lost.

3.4 Flow Control

PCIe uses a credit-based flow control system:

  • The receiver advertises how many credits it has (how much buffer space is available)
  • The transmitter can only send a TLP if it has enough credits to cover that TLP
  • After the receiver processes a TLP and frees buffer space, it sends a Flow Control Update DLLP to return credits to the transmitter

Credit Units

Credit Type Space
Header Credit 1 credit = space for 1 TLP header
Data Credit 1 credit = space for 4 DWORDs of TLP Data

PCIe tracks credits separately for different types of traffic to prevent deadlock.

Credit Pool Used For Completion Expected?
Posed (P) Memory Write, Messages No
Non-Posed (NP) Memory Read Config Read/Write
Completion (Cpl) Completions (responses to NP requests No

Flow Control Initialization
Before any TLPs can be sent, the two sides must initialize flow control. This happens right after the Physical Layer declares the link up:

  • Both sides send InitFC1 DLLPs advertising their initial credits
  • Both sides send InitFC2 DLLPs confirming they received the other side's credits
  • Flow Control is initialized - TLPs can flow now

3.5 DLLPs - Data Link Layer Packets

DLLPs are small control packets used exclusively by the Data Link Layer. They are never seen by the Transaction Layer - they are created and consumed entirely within the DLL.

DLLP Format

+---------------------------------------------+
|Type (1 byte)|Payload (3 bytes)|CRC (2 bytes)|
+---------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

DLLPs are much smaller than TLPs. They are sent in the gaps between TLPs.

DLLP Types

Type Purpose When Sent
ACK Ack received TLPs up to a given Seq# After successfully receiving a TLP
NAK Request retransmission from a given Seq# After detecting a bad TLP
InitFC1 Flow control initialization (phase 1) During link initialization
InitFC2 Flow control initialization (phase 2) During link initialization
UpdateFC Return flow control credits to transmitter After processing a TLP and freeing buffer space

Top comments (0)