Menu

Why move to PCIe Gen 5?

October 04, 2022

Why move to PCIe Gen 5?

By Julia Elbert, Vice President of Engineering

With the availability of PCIe Gen 5 switches from Broadcom, One Stop Systems, Inc. (OSS) is ready to enter the market by supporting multiple IO endpoints like GPUs, NVMe storage devices, and networking cards at the Edge and Data Centers. With the advent of these switches, OSS Engineering designs products that meet the needs of demanding software applications - like artificial intelligence (AI), machine learning (ML) and other high-performance computing. OSS has already engineered numerous multi-slot PCIe Gen 5 backplanes that enable dense configurations of IO inside and outside the server. In general, PCIe 5.0 technology allows for more system topologies than ever before.  Additionally, the feature-rich PCIe 5.0 switch designs help meet the needs of high-speed communications within a server, within a rack, and disaggregated solutions that use the PCIe bus.

Moving to the next generation of PCIe comes with challenges.  However, through years of experience and solution-driven outcomes, OSS Engineering continues to meet those challenges. The PCIe specification doubles the bandwidth and improves power efficiency. And now, with PCIe Gen 5, we move from 16.0 GT/s (gigatransfers per second) to 32.0 GT/s. In other words, bandwidth increases from ~32 GB/s to ~64 GB/s - in each direction. The improved bandwidth works well with multiple IO devices, like NVMe, GPUs, and other high-performance PCIe add-in cards.

Revision Max Data Rate
PCIe 1.0 (2003) 2.5 GT/s 
PCIe 2.0 (2007) 5.0 GT/s 
PCIe 3.0 (2010) 8.0 GT/s 
PCIe 4.0 (2017) 16.0 GT/s 
PCIe 5.0 (2019) 32.0 GT/s

The solutions our customers have come to expect from us require a complicated set of tasks to achieve. Therefore, we must be very thorough in every aspect of our design to ensure our cabled PCIe solutions live up to the rock-solid solutions we are known for in the industry. Understanding our customers' demands, we design all our products to run 24 hours a day, 7 days a week.

There are many critical elements in the design of a PCIe Gen 5 expansion solution, including multiple circuit boards, cables, and connectors. We use a careful selection of board materials, impedance requirements, and matched tracing in our designs to ensure a low risk of signal loss - which degrades performance and adds latency to the system. As we move to PCIe Gen 5, understanding insertion loss is essential in designing efficient systems, so QSFP-DD is the cabling connector we have selected. QSFP-DD is currently the highest quality connector on the market and the correct mechanical fit for our solutions. QSFP-DD industry standard connectors and cables support the target data rate 32GT/s (400Gb/s in a x8) without any changes required.  For example, the connector is already surface-mounted and does not add stubs, as do press-fit connectors (Mini-SAS and CDFP). In addition, QSFP-DD matches the switch native impedance of 100 ohms. See the below chart for comparisons.

Comparison Chart

One of the challenges we discovered in implementing these switches was how to combine their use while supporting side band signaling. Placement and pinout are critical to adhere to the PCI spec, and essential to capturing all the features of a cabled product. The PCIe specification requires the BER (bit error rate) to be less than 10-12 because errors of any kind cause the system to build up latencies and eventually become unreliable, or need to be rebooted. BER is harder to achieve in Gen 5. The circuit designer methodically considers the total insertion loss budget, and accounts for reflections, crosstalk, and power noise that can degrade signal integrity.  Accurate results can only be determined after the design is fabricated, assembled, and tested. For OSS, we have 16 lanes of PCIe to check and focus on in our compliance measurements for each IO add-in PCIe card in our backplanes. As illustrated in the results below, OSS is meeting the requirements of the PCIe Gen 5 in this eye measurement on all 16 lanes of a PCIe slot, which resides 3 meters over a cable outside the server.

All Eyes Graph

Ensuring there are zero errors on a PCIe bus makes for a low latency, simplified switching solution, and is a critical business use for customers in high-speed compute environments. Once topologies are constructed, the data flows directly between connected devices with hardware support enabling high performance with multi-host connectivity, I/O sharing, and DMA direct data transfers.

By migrating to PCIe Gen 5 and using Broadcom switches, engineers can make available all the industry-first features for the most demanding data-intense applications. One Stop Systems is highly qualified and eager to assist our customers in taking advantage of these advancements.

Click the buttons below to share this blog post!

Return to the main Blog page




Leave a comment


Also in One Stop Systems Blog

Tackling the Thermal Challenge of 600W+ Devices in High-Density Computing Systems
Tackling the Thermal Challenge of 600W+ Devices in High-Density Computing Systems

June 02, 2025 2 Comments

 

When the PCI-SIG formally added support for 675W add-in card devices in the PCI Express Card Electromechanical (CEM) specification in August 2023, NVIDIA’s most powerful CEM GPU, the NVIDIA H100 80GB had a maximum power consumption of 350W.  While some devices were starting to push the limits of datacenter thermodynamics – high density systems of many 675W devices seemed like a distant reality.  However, with power constraints uncapped and the need for higher performing GPUs skyrocketing, the industry quickly came out with devices taking full advantage of the new specification capability.  NVIDIA quickly replaced the H100 80GB with the H100 NVL, increasing power density to 400W.  While this small jump was manageable for existing installations, NVIDIA then dove all-in with the H200 NVL released in late 2024 at 600W.  The rapid transition from 350W to 600W has put power and cooling technologies in the spotlight in a race to solve this next generation challenge.

Continue Reading

The Future of Transportation: Will Autonomous Trucks Ever Make the Driver Obsolete?
The Future of Transportation: Will Autonomous Trucks Ever Make the Driver Obsolete?

April 14, 2025 2 Comments

The advent of technology has always brought about significant changes to various industries, and the transportation sector is no exception. Among the most transformative innovations in recent years is the development of autonomous vehicles, particularly trucks. The potential for autonomous trucks to revolutionize freight transport is immense, raising the fundamental question: will these technological advancements make human drivers obsolete? To explore this question, we must consider the current state of autonomous driving technology, the economic implications, and the societal impact of removing human drivers from the equation.

Continue Reading

Advantages and Disadvantages of Implementing AI Inference Nodes on Soldiers
Advantages and Disadvantages of Implementing AI Inference Nodes on Soldiers

January 15, 2025 2 Comments

The integration of artificial intelligence (AI) into military operations has revolutionized battlefield strategies, decision-making, and operational efficiency. Among these advancements, AI inference nodes deployed directly on soldiers represents a cutting-edge innovation. These nodes, compact computational devices, enable real-time AI processing and analytics, empowering soldiers with enhanced situational awareness, decision support, and operational effectiveness. However, such technology also brings challenges, particularly in power management, size, and weight constraints. This blog delves into the advantages and disadvantages of implementing AI inference nodes on soldiers, focusing on these critical aspects.

Continue Reading

You are now leaving the OSS website