← Back
Home › Systems › Hardware and datacenter

ai-accelerator

AI accelerator

TPU, Trainium

SystemTechniques · 7Primitives · 33 ai-accelerator Vliw static schedulinghardware-os · medium: VLIW bundles: compiler-scheduled parallel slots, predication and exposed latenciesTensor compiler schedulingml-training · medium: Tile, fuse and lay out tensor ops for a target (XLA, TVM, Triton schedules), plus autotuningInstruction schedulingcompilers · medium: List scheduling over dependence DAGs with latency models; software pipelining of loopsRegister allocationcompilers · medium: Map unlimited virtual registers onto a few machine registers, spilling the rest to the stack; linear scan is fast, graph coloring and splitting allocators give better code.›Kernel autotuningml-training · medium: Searching tile sizes and launch parameters per shape/hardware, caching the best configsDataflow analysiscompilers · medium: Compute a fact per program point (live variables, reaching definitions, available expressions) by iterating transfer functions over the CFG to a fixed point.›Graph coloringgraphs · medium: Greedy/DSatur colouring for conflict-free assignment such as radio channelsKernel fusionml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.›Tiled matrix multiplicationml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.›Roofline modelhardware-os · low: Bound a kernel's speed by min(peak FLOP/s, arithmetic intensity × memory bandwidth) to know whether to optimize compute or data movement.›Double buffering dmahardware-os · low: Overlap compute with DMA into ping-pong scratchpad buffers on accelerators and DSPsDram organizationhardware-os · low: Channels, ranks, banks, rows; row-buffer hits, refresh, timing parameters; DDR5 and HBMNetwork on chiphardware-os · low: On-chip mesh and ring interconnects: routers, virtual channels, deadlock-free routingSystolic matrix unitshardware-os · low: Systolic arrays and tensor cores: dataflow of operands, tile shapes, accumulator precisionFlashAttentionml-training · low: Exact attention computed tile by tile in on-chip SRAM with online softmax, so the N×N score matrix never touches HBM; 2-4x faster, O(N) memory.›Ring all-reduceml-training · low: Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.›Structured sparsityml-training · low: 2:4 and block sparsity patterns that hardware can accelerate; pruning to fit themWeight quantizationml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.›Network topologiesnetworking · low: Fat-tree and Clos, dragonfly, torus and rail-optimized topologies: bisection bandwidth and diameterLow precision number formatsnumerics · low: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scalingHamming codessignal-processing · low: Hamming and SECDED block codes for single-error correction in links and memoryTensor layout transformsml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiencyTensor memory planningml-training · low: Static buffer reuse from tensor lifetimes so a graph runs in a fixed arena with minimal peak memory+ 17 more primitives

Similar systems

Share the most building blocks with this one.

gpu 29 %

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

TrainiumAmazonTPUAlphabet