← Back
Home › Systems › Frontier AI research

multi-datacenter-training

frontier pretraining across datacenters

Gemini, GPT-5-class runs

SystemTechniques · 20Primitives · 72 multi-datacenter-traini… Power oversubscription cappingcloud · medium: Oversubscribe datacenter power with telemetry and fast capping (RAPL, priority-based throttling)Gang schedulingdistributed · medium: All-or-nothing co-scheduling of a job's tasksDistributed checkpointingml-training · medium: Sharded, asynchronous checkpoint save/load of model and optimizer state, with resharding to new world sizesFederated averagingml-training · medium: FedAvg: clients train locally, server averages weighted updates; handling non-IID data and stragglersGpu health checksml-training · medium: Detecting bad GPUs/links before and during jobs: ECC/XID errors, NCCL tests, throughput probes, drainingDatacenter congestion controlnetworking · medium: ECN-based schemes for RDMA and DC TCP: DCQCN, DCTCP, HPCC, plus PFC lossless pitfallsIn network aggregationnetworking · medium: Reduce gradients or data inside switches (SHARP, SwitchML) to cut allreduce trafficCollective communicationml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.›ZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›Gradient compressionml-training · low: Reducing communicated updates by quantization, top-k sparsification and error feedbackRdmanetworking · low: RDMA verbs over InfiniBand/RoCE: queue pairs, memory registration, one-sided opsPhi accrual failure detectordistributed · low: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.›Speculative executiondistributed · low: Launch backup copies of straggler tasks and keep the first resultPipeline parallelismml-training · low: Split the model's layers into stages on different GPUs and stream micro-batches through them; 1F1B scheduling bounds activation memory, the bubble costs (p−1)/m.›Ring all-reduceml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.›Tensor parallelismml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.›BBR congestion controlnetworking · low: Model-based congestion control that paces at the measured bottleneck bandwidth and caps inflight near one BDP, instead of reacting to loss.›Fixed-rate control loopembedded · low: Run a controller at an exact period using absolute deadlines and a fixed dt, so jitter never accumulates and the math stays valid.›Power management dvfshardware-os · low: DVFS governors, P- and C-states, race-to-idle, thermal throttlingDistributed barrierdistributed · low: Rendezvous/barrier for a dynamic worker group: join, agree on membership and rank assignment, restart on changeObject storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Atomic file replacementhardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updatesTensor layout transformsml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiency+ 13 more techniques+ 56 more primitives

Similar systems

Share the most building blocks with this one.

training-fabric 35 %gigawatt-ai-campus 29 %elastic-training 25 %rack-scale-system 19 %llm-pretraining 18 %