Summary
Highlights
PCIe Gen 6 is already making an impact, not in CPUs, but in the connection between networking and GPUs. This is considered abnormal but driven by the need for less overhead in communication channels. The speakers anticipate that on-chip photonics will be an important development for consumers due to its potential for cost-effectiveness and increased efficiency compared to traditional electrical signal transmission.
PCIe continues to be the future, with changes in how PCIe lanes are utilized. It is now subsuming networking and communication channels, making Ethernet look more like PCIe. Modern networking components, such as 800 gigabit and 1.6 terabyte devices, are costly but essential for high-speed data transfer in data centers.
The convergence of compute, networking, and PCIe is crucial for modern servers. Examples include Broadcom controllers that manage PCIe lanes and microtic switches with RDMA capabilities, allowing GPUs to directly access storage through a networking fabric that resembles PCIe. Various adapters and cards illustrate how PCIe lanes are being fanned out and adapted for different storage and networking formats.
PCIe Gen 6 is already in use for GPU-to-GPU and GPU-to-networking communication in enterprise settings, offering 1.6 terab speeds. The shift towards on-chip photonics is expected to make its way to the consumer market, primarily through optical connections for peripherals like NVMe drives, offering benefits such as lower power consumption, reduced heat, and improved signal integrity.
The discussion touches on the viability of repurposing old enterprise hardware for consumer use. While some memory (like LPDDR5/5X and HBM) is often soldered or in incompatible formats, other components, particularly older Xeon processors and data center GPUs like V100s, can be re-worked by manufacturers, especially in regions like China, to create affordable, high-performance systems for enthusiasts and academics.
PCIe is expanding its reach, even into microcontrollers like the Raspberry Pi. The continued development of PCIe as an interconnect is simplifying networking and offering more affordable high-speed options. The speakers express optimism that enterprise gear, such as V100 GPUs, will become more readily available for reuse, especially for academic and enthusiast purposes, highlighting the benefits of open standards and the societal need for accessible technology.
The surprisingly low cost of 100 gigabit optics opens possibilities for consumers to set up high-speed connections for home networks and NAS setups without needing expensive switches. While latency might be slightly worse, enterprise efforts to reduce it are pushing these technologies closer to native PCIe connections. The importance of improved RDMA plumbing in Linux is also highlighted for broader adoption of high-speed networking.
The widespread adoption of PCIe is beneficial as it moves away from proprietary interfaces common in the past, offering greater compatibility and future-proofing. The shift towards open standards, though driven by enterprise needs, inadvertently benefits consumers by making cutting-edge technologies like optical PCIe accessible as they become the only viable options for certain applications.