mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Hardware and datacenter
training-fabric
GPU training network fabric
InfiniBand, RoCE Clos
System
Techniques · 4
Primitives · 36
training-fabric
Datacenter congestion control
networking · medium: ECN-based schemes for RDMA and DC TCP: DCQCN, DCTCP, HPCC, plus PFC lossless pitfalls
In network aggregation
networking · medium: Reduce gradients or data inside switches (SHARP, SwitchML) to cut allreduce traffic
Active queue management
networking · medium: RED, CoDel and PIE: drop or ECN-mark early to keep switch and router queues short
Match action pipelines
networking · medium: RMT switch pipelines: parsed headers flow through match-action stages; P4 programming model
Rdma
networking · low: RDMA verbs over InfiniBand/RoCE: queue pairs, memory registration, one-sided ops
Collective communication
ml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.
›
DMA and IOMMU
hardware-os · low: How devices move data by direct memory access, and how an IOMMU gives each device its own address space, faults on stray accesses and lets user space or guests own a device safely.
›
Numa aware allocation
hardware-os · low: Allocate memory and pin threads on the local NUMA node; first-touch and interleave policies
Pipeline parallelism
ml-training · low: Split the model's layers into stages on different GPUs and stream micro-batches through them; 1F1B scheduling bounds activation memory, the bubble costs (p−1)/m.
›
Ring all-reduce
ml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.
›
Tensor parallelism
ml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.
›
ZeRO and FSDP sharding
ml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.
›
Ecmp
networking · low: Equal-cost multipath: hash flows across next hops; resilient hashing on changes
Kernel-bypass networking
networking · low: User space (DPDK, AF_XDP) or an in-driver eBPF hook (XDP) handles packets straight from NIC rings, skipping sockets, for 10-40 Mpps per core.
›
Network topologies
networking · low: Fat-tree and Clos, dragonfly, torus and rail-optimized topologies: bisection bandwidth and diameter
Active path probing
networking · low: Continuous loss/latency/jitter probes per WAN path to drive path selection
Credit based flow control
networking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)
Deterministic fixed-point math
game-networking · low: Bit-identical simulation math across CPUs, compilers and builds: integer fixed-point formats, the float traps (FMA, x87, fast-math, libm), deterministic trig, sqrt and RNG.
›
Selective repeat arq
networking · low: Selective-repeat ARQ with windows, status reports and in-order reassembly
Fair queuing
networking · low: Deficit round robin / weighted fair queuing to share capacity among tenants or flows
+ 20 more primitives
System
Techniques · 4
Primitives · 36
training-fabric
Datacenter congestion control
networking · medium: ECN-based schemes for RDMA and DC TCP: DCQCN, DCTCP, HPCC, plus PFC lossless pitfalls
In network aggregation
networking · medium: Reduce gradients or data inside switches (SHARP, SwitchML) to cut allreduce traffic
Active queue management
networking · medium: RED, CoDel and PIE: drop or ECN-mark early to keep switch and router queues short
Match action pipelines
networking · medium: RMT switch pipelines: parsed headers flow through match-action stages; P4 programming model
Rdma
networking · low: RDMA verbs over InfiniBand/RoCE: queue pairs, memory registration, one-sided ops
Collective communication
ml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.
›
DMA and IOMMU
hardware-os · low: How devices move data by direct memory access, and how an IOMMU gives each device its own address space, faults on stray accesses and lets user space or guests own a device safely.
›
Numa aware allocation
hardware-os · low: Allocate memory and pin threads on the local NUMA node; first-touch and interleave policies
Pipeline parallelism
ml-training · low: Split the model's layers into stages on different GPUs and stream micro-batches through them; 1F1B scheduling bounds activation memory, the bubble costs (p−1)/m.
›
Ring all-reduce
ml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.
›
Tensor parallelism
ml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.
›
ZeRO and FSDP sharding
ml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.
›
Ecmp
networking · low: Equal-cost multipath: hash flows across next hops; resilient hashing on changes
Kernel-bypass networking
networking · low: User space (DPDK, AF_XDP) or an in-driver eBPF hook (XDP) handles packets straight from NIC rings, skipping sockets, for 10-40 Mpps per core.
›
Network topologies
networking · low: Fat-tree and Clos, dragonfly, torus and rail-optimized topologies: bisection bandwidth and diameter
Active path probing
networking · low: Continuous loss/latency/jitter probes per WAN path to drive path selection
Credit based flow control
networking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)
Deterministic fixed-point math
game-networking · low: Bit-identical simulation math across CPUs, compilers and builds: integer fixed-point formats, the float traps (FMA, x87, fast-math, libm), deterministic trig, sqrt and RNG.
›
Selective repeat arq
networking · low: Selective-repeat ARQ with windows, status reports and in-order reassembly
Fair queuing
networking · low: Deficit round robin / weighted fair queuing to share capacity among tenants or flows
Packet header parsing
networking · low: Parse graph state machines for Ethernet, IP, TCP and UDP headers in hardware and software
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Exponential smoothing (ETS, Holt-Wi…
statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.
›
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Longest prefix match
networking · low: IP prefix lookup structures (DIR-24-8, Poptrie, LC-trie) for routes, ACLs and geo-IP
Tcam ternary matching
networking · low: TCAM ternary match tables: priorities, range expansion, power and capacity trade-offs
False sharing
concurrency · low: Independent variables written by different cores on one cache line force the line to bounce between cores; pad or regroup hot data to fix it.
›
Memory ordering
concurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.
›
SPSC ring buffer
data-structures · low: Bounded wait-free queue for exactly one producer and one consumer thread, built from an array and two indices with acquire/release ordering.
›
CPU caches and the memory hierarchy
hardware-os · low: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.
›
Gpu memory coalescing banking
hardware-os · low: Coalesced global loads, shared memory bank conflicts, L2 and HBM bandwidth on GPUs
Huge pages
hardware-os · low: 2 MB/1 GB pages to cut TLB misses; THP vs explicit hugetlbfs
CUDA and GPU languages
languages · low: Programming GPUs: thousands of lockstep threads, bound by memory bandwidth; libraries first, Triton for fused kernels, CUDA C++ for peak speed.
›
AdamW
ml-training · low: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.
›
Mixed-precision training
ml-training · low: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.
›
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Ebpf xdp
networking · low: EBPF programs at the XDP/tc hooks for in-kernel packet processing at line rate
Receive side scaling
networking · low: NIC hashing of flows to RX queues (RSS) and symmetric hashing for per-core processing
Floating-point error
numerics · low: How IEEE 754 rounding error accumulates and how to contain it: compensated (Kahan/Neumaier) and pairwise summation, error-free transformations with FMA, ULP comparisons.
›
Low precision number formats
numerics · low: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scaling
Copy for AI
Similar systems
Share the most building blocks with this one.
multi-datacenter-training
35 %
mpi-cluster
30 %
scale-up-fabric
27 %
rack-scale-system
23 %
gigawatt-ai-campus
21 %