← Back
Home › Systems › AI and ML infrastructure

elastic-training

fault-tolerant elastic distributed training

TorchElastic

SystemTechniques · 6Primitives · 32 elastic-training Raftdistributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.›Data loading pipelinesmachine-learning · medium: Turn files on disk into shuffled, sharded, resumable GPU batches fast enough that the accelerator never waits.›Distributed checkpointingml-training · medium: Sharded, asynchronous checkpoint save/load of model and optimizer state, with resharding to new world sizesGpu health checksml-training · medium: Detecting bad GPUs/links before and during jobs: ECC/XID errors, NCCL tests, throughput probes, drainingErasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationState-machine snapshotsdistributed · medium: Periodically save the state a log has produced at index i so the log prefix can be deleted; recovery and catch-up become load snapshot plus replay suffix.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Collective communicationml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.›ZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›Distributed barrierdistributed · low: Rendezvous/barrier for a dynamic worker group: join, agree on membership and rank assignment, restart on changePhi accrual failure detectordistributed · low: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.›Ring all-reduceml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›mmap and the page cachehardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Sequence packingml-training · low: Packing variable-length sequences into fixed-size batches with attention masks to avoid padding wastePRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Atomic file replacementhardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updatesTensor layout transformsml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiencyEdac memory scrubbingembedded · low: ECC/EDAC memory with periodic scrubbing to correct radiation-induced bit flipsTiled matrix multiplicationml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.›+ 16 more primitives

Similar systems

Share the most building blocks with this one.

gpu-job-scheduler 25 %multi-datacenter-training 25 %distributed-fs 17 %