By Julia Elbert, Vice President of Engineering
With the availability of PCIe Gen 5 switches from Broadcom, One Stop Systems, Inc. (OSS) is ready to enter the market by supporting multiple IO endpoints like GPUs, NVMe storage devices, and networking cards at the Edge and Data Centers. With the advent of these switches, OSS Engineering designs products that meet the needs of demanding software applications - like artificial intelligence (AI), machine learning (ML) and other high-performance computing. OSS has already engineered numerous multi-slot PCIe Gen 5 backplanes that enable dense configurations of IO inside and outside the server. In general, PCIe 5.0 technology allows for more system topologies than ever before. Additionally, the feature-rich PCIe 5.0 switch designs help meet the needs of high-speed communications within a server, within a rack, and disaggregated solutions that use the PCIe bus.
Moving to the next generation of PCIe comes with challenges. However, through years of experience and solution-driven outcomes, OSS Engineering continues to meet those challenges. The PCIe specification doubles the bandwidth and improves power efficiency. And now, with PCIe Gen 5, we move from 16.0 GT/s (gigatransfers per second) to 32.0 GT/s. In other words, bandwidth increases from ~32 GB/s to ~64 GB/s - in each direction. The improved bandwidth works well with multiple IO devices, like NVMe, GPUs, and other high-performance PCIe add-in cards.
The solutions our customers have come to expect from us require a complicated set of tasks to achieve. Therefore, we must be very thorough in every aspect of our design to ensure our cabled PCIe solutions live up to the rock-solid solutions we are known for in the industry. Understanding our customers' demands, we design all our products to run 24 hours a day, 7 days a week.
There are many critical elements in the design of a PCIe Gen 5 expansion solution, including multiple circuit boards, cables, and connectors. We use a careful selection of board materials, impedance requirements, and matched tracing in our designs to ensure a low risk of signal loss - which degrades performance and adds latency to the system. As we move to PCIe Gen 5, understanding insertion loss is essential in designing efficient systems, so QSFP-DD is the cabling connector we have selected. QSFP-DD is currently the highest quality connector on the market and the correct mechanical fit for our solutions. QSFP-DD industry standard connectors and cables support the target data rate 32GT/s (400Gb/s in a x8) without any changes required. For example, the connector is already surface-mounted and does not add stubs, as do press-fit connectors (Mini-SAS and CDFP). In addition, QSFP-DD matches the switch native impedance of 100 ohms. See the below chart for comparisons.
One of the challenges we discovered in implementing these switches was how to combine their use while supporting side band signaling. Placement and pinout are critical to adhere to the PCI spec, and essential to capturing all the features of a cabled product. The PCIe specification requires the BER (bit error rate) to be less than 10-12 because errors of any kind cause the system to build up latencies and eventually become unreliable, or need to be rebooted. BER is harder to achieve in Gen 5. The circuit designer methodically considers the total insertion loss budget, and accounts for reflections, crosstalk, and power noise that can degrade signal integrity. Accurate results can only be determined after the design is fabricated, assembled, and tested. For OSS, we have 16 lanes of PCIe to check and focus on in our compliance measurements for each IO add-in PCIe card in our backplanes. As illustrated in the results below, OSS is meeting the requirements of the PCIe Gen 5 in this eye measurement on all 16 lanes of a PCIe slot, which resides 3 meters over a cable outside the server.
Ensuring there are zero errors on a PCIe bus makes for a low latency, simplified switching solution, and is a critical business use for customers in high-speed compute environments. Once topologies are constructed, the data flows directly between connected devices with hardware support enabling high performance with multi-host connectivity, I/O sharing, and DMA direct data transfers.
By migrating to PCIe Gen 5 and using Broadcom switches, engineers can make available all the industry-first features for the most demanding data-intense applications. One Stop Systems is highly qualified and eager to assist our customers in taking advantage of these advancements.
Click the buttons below to share this blog post!
Comments will be approved before showing up.
Get the latest news, product releases, upcoming events, blog posts and more!