ALLPCB
Since the introduction of the Transformer and the rise of ChatGPT in 2023, it has become clear that larger model parameter counts correlate with improved performance following scaling laws. As model parameter counts grow into the tens of billions and beyond, language understanding, logical reasoning, and problem analysis capabilities improve rapidly. Concurrent with increases in model size and performance, the network requirements for training these large models change significantly.
Large-scale distributed training typically combines data parallelism, pipeline parallelism, and tensor parallelism. Each parallelism mode requires collective communication across multiple devices. Training is often synchronous, so collective operations must complete across machines and GPUs before the next iteration can proceed. Designing cluster network architectures that provide low latency and high throughput for inter-node communication is critical to reduce synchronization overhead and increase the GPU compute time fraction (GPU compute time / total training time). The following sections analyze network requirements from the perspectives of scale, bandwidth, latency, stability, and deployment automation.
1. Ultra-large-scale networking
AI compute demand has grown exponentially, and model sizes have expanded dramatically over the past decade. Current very large models reach hundreds of billions to trillions of parameters. Training such models requires extremely high compute and very large memory capacity. For example, a 1T-parameter model stored at 16-bit precision consumes about 2 TB of parameter storage. Activations from forward passes, gradients from backward passes, and optimizer state also require storage, and intermediate variables grow during an iteration. A training run using the Adam optimizer can peak at roughly seven times the model parameter size in intermediate memory. Such high memory demands mean tens to hundreds of GPUs are required to store and train a single large model.
Having many GPUs is not sufficient by itself; appropriate parallelism strategies are essential to achieve efficient training. Large models typically use data parallelism, pipeline parallelism, and tensor parallelism concurrently. Training models at the hundreds-of-billions to trillions scale requires clusters of thousands of GPUs. While cloud data centers may interconnect tens of thousands of servers, interconnecting thousands of GPUs is more challenging because network capability must be tightly matched to compute capability. Typical CPU-based cloud workloads use 10–100 Gbps networking with TCP. Large-scale GPU training demands 100–400 Gbps interconnects and relies on RDMA to reduce latency and improve throughput.
Key issues to consider for high-performance networking across thousands of GPUs include:
- Failure modes in large RDMA deployments, such as head-of-line blocking and PFC deadlock storms
- Network performance optimization, including more effective congestion control and load balancing
- NIC connection capacity: hardware limits on a single host and how to scale to thousands of RDMA queue pair connections
- Network topology choice: whether traditional fat-tree topologies remain optimal or if high-performance computing topologies like torus or dragonfly should be considered
2. Ultra-high bandwidth
Collective communication operations both inside and between servers generate large volumes of data in large-model training. For intra-node GPU communication, model-parallel AllReduce traffic for a model at the hundred-billion parameter scale can reach the order of hundreds of gigabytes, so intra-server GPU interconnect bandwidth and topology are critical to flow completion time. High-speed GPU interconnects avoid repeated copies through CPU memory during GPU communication.
For inter-node GPU communication, pipeline, data, and tensor parallel patterns require different collective operations. Some collective transfers also reach the order of hundreds of gigabytes and produce many-to-one and one-to-many traffic patterns simultaneously. This places high demands on per-port bandwidth, the number of available links per node, and the aggregate network capacity. The PCIe bus between GPU and NIC also limits achievable throughput: for example, PCIe 3.0 x16 provides about 16 GB/s per direction, so a 200 Gbps (25 GB/s) network port cannot be fully utilized when attached to a PCIe 3.0 x16 host bus.
3. Ultra-low latency and low jitter
Network latency consists of static and dynamic components. Static latency includes serialization delay, device forwarding delay, and optoelectronic conversion delay; it is determined by forwarding-chip capability and physical distance and is largely fixed for a given topology and traffic pattern. Dynamic latency, which typically has a greater impact on performance, includes switch queuing delay and retransmission delay caused by congestion and packet loss.
For example, analysis of training at the 175-billion-parameter scale indicates that if dynamic latency increases from 10 microseconds to 1000 microseconds, the GPU compute-time fraction can drop by nearly 10%. When packet loss reaches 0.1% (one in a thousand), GPU compute-time fraction can fall by about 13%. At 1% packet loss, the GPU compute-time fraction can fall below 5%. Reducing communication latency and increasing network throughput are therefore central to fully utilizing compute resources in large-scale training clusters.
Latency jitter also degrades training efficiency. Collective operations can be decomposed into multiple parallel P2P communications; for example, a Ring AllReduce across N nodes consists of 2*(N-1) communication steps, and each step requires all nodes to complete their P2P transfers in parallel before the step finishes. If network fluctuations cause one P2P flow's flow completion time to grow substantially, that slower flow becomes the weakest link and delays the whole step. Jitter-induced imbalance therefore reduces collective efficiency and overall training throughput.
4. Ultra-high stability and availability
Since the Transformer era, large models have evolved rapidly. Model sizes have grown from tens of millions to hundreds of billions of parameters. Cluster compute capacity determines training time: a single V100 GPU would need roughly 335 years to train GPT-3, while a cluster of 10,000 V100 GPUs with perfect linear scaling would take on the order of 12 days. Network availability underpins cluster stability. A single network node failure can affect tens or more compute nodes, reducing effective cluster capacity. Network performance fluctuations are particularly challenging because the network is a shared resource and is harder to isolate than an individual compute node. Performance volatility can reduce utilization across all compute resources, so maintaining a stable and efficient network is a critical operational goal for large-model training clusters.
During training, failures may require fault-tolerant replacement or elastic scaling. Changes in the set of participating nodes can render the current communication layout suboptimal, necessitating job reshuffling and rescheduling to restore efficiency. Some network faults, such as silent packet loss, are hard to predict. These faults can degrade collective operation efficiency and trigger communication library timeouts, causing long stalls and severely impacting training progress. Collecting fine-grained metrics such as per-flow throughput and packet loss enables faster fault mitigation and can limit self-healing times to the order of seconds.
5. Automated network deployment and operations
Lossless or intelligent RDMA-based networks with advanced congestion control require many diverse and complex configuration parameters. Any misconfiguration can degrade performance or produce unexpected behaviors. Studies indicate that misconfiguration causes the majority of high-performance network outages, driven by the large number of NIC parameters that vary with hardware generation, workload type, and NIC model. Large training clusters further increase configuration complexity.
Automated deployment and configuration reduce operational risk and improve cluster reliability and efficiency. Automation should support parallel configuration across many hosts, automatic selection of congestion-control parameters, and profile-driven configuration choices based on NIC and workload type. In complex environments, rapid and accurate fault localization during runtime preserves overall training efficiency. Automated fault detection can quickly narrow problem scope, notify administrators with precise diagnostics, reduce troubleshooting costs, and help identify root causes and remediation steps.
