← Back
Home › Systems › Hardware and datacenter

scale-up-fabric

GPU scale-up interconnect fabric

NVIDIA NVLink/NVSwitch, AMD Infinity Fabric, UALink

SystemTechniques · 5Primitives · 25 scale-up-fabric Memory semantic interconnecthardware-os · medium: Load/store fabrics (CXL, NVLink-style) giving hosts cache-coherent or pooled remote memory with address windows and device-managed coherenceGpu health checksml-training · medium: Detecting bad GPUs/links before and during jobs: ECC/XID errors, NCCL tests, throughput probes, drainingIn network aggregationnetworking · medium: Reduce gradients or data inside switches (SHARP, SwitchML) to cut allreduce trafficTopology aware placementsystems · medium: Place work by NUMA, rack, zone and GPU interconnect topologyMatch action pipelinesnetworking · medium: RMT switch pipelines: parsed headers flow through match-action stages; P4 programming modelCollective communicationml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.›Network topologiesnetworking · low: Fat-tree and Clos, dragonfly, torus and rail-optimized topologies: bisection bandwidth and diameterMemory orderingconcurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.›DMA and IOMMUhardware-os · low: How devices move data by direct memory access, and how an IOMMU gives each device its own address space, faults on stray accesses and lets user space or guests own a device safely.›Virtual memory and the TLBhardware-os · low: How page tables and TLBs translate every address, what TLB reach means, and when 2 MiB or 1 GiB huge pages pay off on Linux.›Ring all-reduceml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.›Tensor parallelismml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.›Credit based flow controlnetworking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)Adaptive equalizationsignal-processing · low: LMS/RLS equalizer filters that invert channel distortion from training or decisionsForward error correctionsignal-processing · low: Add redundancy (LDPC, Reed-Solomon, XOR parity) so receivers repair loss without retransmitLine coding scramblingsignal-processing · low: Scramblers and line codes (8b/10b, 64b/66b) for DC balance and clock recoveryEdac memory scrubbingembedded · low: ECC/EDAC memory with periodic scrubbing to correct radiation-induced bit flipsTiled matrix multiplicationml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.›Robust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›Deterministic fixed-point mathgame-networking · low: Bit-identical simulation math across CPUs, compilers and builds: integer fixed-point formats, the float traps (FMA, x87, fast-math, libm), deterministic trig, sqrt and RNG.›Selective repeat arqnetworking · low: Selective-repeat ARQ with windows, status reports and in-order reassembly+ 9 more primitives

Similar systems

Share the most building blocks with this one.

rack-scale-system 42 %training-fabric 27 %multi-datacenter-training 18 %mpi-cluster 17 %gigawatt-ai-campus 16 %