By Ray DeMoss, Support Manager

If you work in high-performance compute (HPC) or system support, running lspci -vvv is almost muscle memory. It is your first window into the PCIe bus: checking negotiated speed, link width, and device status. When a system in an air-conditioned laboratory drops bandwidth, lspci usually tells you enough to start troubleshooting.

But when compute platforms move out of the data center and into harsh edge environments—like inside autonomous ground vehicles, mobile command centers, or aircraft—lspci quickly hits its limits.

At PCIe Gen5 (32 GT/s) and Gen6 (64 GT/s), electrical tolerances shrink to nanosecond margins. When a high-density GPU or NVMe storage array suddenly negotiates down to Gen1 speeds or drops from a x16 to a x4 link width, lspci confirms that the failure happened, but it cannot tell you why. Was it a thermal drift changing trace impedance? Connector fretting from chassis vibration? Or did the system simply step down to conserve power?

To keep high-throughput edge AI systems operational without bringing expensive oscilloscopes into the field, support and test engineers need to move past standard status commands and utilize in-band hardware telemetry.  In this article, we demonstrate how to leverage native Linux utilities like pcicrawler, kernel Advanced Error Reporting (AER) logs, and software-driven Receiver Eye Margining (pcilmr) to diagnose physical layer signal degradation, isolate link down-training, and maintain peak PCIe Gen4/Gen5/Gen6 fabric performance at the edge. 

Down-Training vs. ASPM: Distinguishing Faults from Power Saving

One of the most common false alarms in field diagnostics occurs when a high-speed PCIe endpoint reports running at 2.5 GT/s (PCIe Gen1). Before disassembling a rugged chassis or replacing cables, you must determine whether the link is physically degraded or simply idling.

Active State Power Management (ASPM) intentionally drops PCIe link speeds during idle periods to reduce power consumption and thermal load. However, under physical stress, the PCIe Link Training and Status State Machine (LTSSM) may encounter bit errors and permanently down-train the link to maintain a stable connection.

To isolate the root cause in the field, force the kernel's ASPM policy to performance mode:

echo performance > /sys/module/pcie_aspm/parameters/policy

If putting the system under load restores the link to maximum rate, ASPM was active. If the device remains locked at degraded speeds or narrower widths, you are dealing with physical layer signal degradation.

Automated Bus Inspection Beyond Standard Command Outputs

Parsing massive PCI bus trees manually during field testing is inefficient. Open-source utilities like pcicrawler allow engineers to dump the complete PCIe topology into structured JSON. By filtering current link performance against maximum capable device specs, down-trained endpoints can be flagged immediately:

pcicrawler -j | jq -r '.[] | select(.cur_speed != .capable_speed or .cur_width != .capable_width) | "\(.addr)\t\(.cur_speed) x\(.cur_width)\t\(.capable_speed) x\(.capable_width)"' | while read -r addr cur cap; do
  name=$(lspci -s "$addr" | cut -d' ' -f2- | cut -c1-65)
  printf "%-14s %-67s %-12s %-12s\n" "$addr" "$name" "$cur" "$cap"
done

Running this check provides a clear breakdown of physical topology versus active link status: Notice how 0000:a0:03.1 is running at x8 instead of its capable x16, while NVMe controller 0000:e2:00.0 is constrained to 8 GT/s (Gen3) on a Gen4-capable path.

Decoding Physical Integrity with Advanced Error Reporting (AER)

When physical layer signal degradation occurs, the PCIe fabric attempts hardware-level retries before dropping a link entirely. Modern systems log these events through Advanced Error Reporting (AER) rather than legacy parallel PCI SERR/PERR events.  Monitoring native AER logs reveals whether hardware is actively struggling with signal noise or packet loss:

dmesg -T | grep -iE "AER|PCIe|Correctable|Uncorrectable"

A high frequency of Correctable Errors (such as Receiver Errors or Bad TLPs) indicates that the link is recovering automatically, but physical margins are failing. Catching these errors in telemetry allows support teams to flag failing traces or unseated interconnects before an uncorrectable fatal error crashes the application.

Software-Driven Eye Margining in the Field with pcilmr

Defined formally under the PCI-SIG PCIe Base Specification (Section 4.2.13, Lane Margining at the Receiver), modern Gen4, Gen5, and Gen6 silicon integrate internal hardware samplers that support non-destructive testing directly over the bus. Historically, evaluating an 'eye diagram' (measuring signal jitter and voltage margins) required physical access to high-speed bench oscilloscopes, but using in-band utilities like pcilmr, engineers can sweep voltage and timing margins directly at the edge .

Using internal sampler offsets inside modern PCIe PHYs, utilities like pcilmr executes non-destructive timing and voltage sweeps directly over the bus.  To run a system-wide margining sweep on active high-speed links:

pcilmr --full | tee /tmp/eye_data.log

Evaluating real field telemetry against PCI-SIG specification thresholds reveals lane-by-lane physical variance:

Link 60:3.3 -> 62:0.0 (Gen 4 16.0 GT/s):
Spec Minimums: EW >= 18.75 ps | EH >= 15.00 mV

Rx(F) Lane 0: Perfect (W 49.2% UI - 30.75ps, H 247.3 mV)
Rx(F) Lane 1: Perfect (W 47.6% UI - 29.76ps, H 229.2 mV)
Rx(F) Lane 2: Perfect (W 50.8% UI - 31.75ps, H 223.2 mV)
Rx(F) Lane 3: Perfect (W 44.4% UI - 27.78ps, H 150.8 mV)

While all four lanes comfortably clear specification minimums, Lane 3 exhibits an Eye Height of 150.8 mV compared to 247.3 mV on Lane 0. In-band margining flags this attenuation trend early, giving technicians visibility into trace aging, impedance changes, or thermal stress long before a hard fault occurs.

The OSS Edge Advantage

Software tools like pcilmr, pcicrawler, and kernel AER diagnostics provide essential visibility when servicing deployed systems. However, avoiding downtime at the edge ultimately depends on system architecture.

While these utilities are often considered obscure developer tools, industry engineering teams rely on them heavily for automated fleet health. Meta’s infrastructure team developed pcicrawler to automate the detection of link down-training across data center hardware, and the Linux kernel community introduced pcilmr precisely so field and validation engineers could evaluate high-speed receiver margins directly in-band without bench instruments.  This makes them ideal for practical applications at the edge.

At One Stop Systems (OSS), our rugged compute platforms and high-speed expansion systems are engineered specifically to eliminate fabric bottlenecks under harsh conditions. By pairing thermal management designed for 35°C ambient environments with high-quality retimer topologies and shock-isolated chassis designs, OSS ensures PCIe signal eye openings remain open, and delivers maximum compute throughput where it matters most.