Neural network training & inference

The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.

Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.

Train a network to undo Gaussian noise at every noise level, then generate by integrating from pure noise to data; the standard for images, video and audio.

Record the forward computation on a tape, then sweep it backwards once to get the gradient of a scalar loss for all inputs.

Gradient descent plus a decaying velocity buffer that averages minibatch noise and speeds up progress along shallow, consistent directions.

Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.

Ramp the learning rate up linearly for the first ~1% of steps, then decay it along a half cosine to ~10% of peak.

Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.

Save only some activations in the forward pass and recompute the rest during backward, trading about 33% extra compute for O(√n) activation memory.

Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.

Exact attention computed tile by tile in on-chip SRAM with online softmax, so the N×N score matrix never touches HBM; 2-4x faster, O(N) memory.

Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.

The vendor-neutral GPU launch model of Metal, Vulkan and WebGPU: a grid of workgroups, shared memory, barriers and subgroups, driven by recorded command buffers.

Sum a buffer across N workers via reduce-scatter then all-gather around a ring; each link carries 2(N−1)/N of the buffer, which is bandwidth-optimal.

Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.

Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.

Split the model's layers into stages on different GPUs and stream micro-batches through them; 1F1B scheduling bounds activation memory, the bubble costs (p−1)/m.

The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.

Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.

Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.

Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.

A cheap draft proposes γ tokens, the big model verifies them in one pass, and rejection sampling keeps the target's exact distribution at 2-3x speed.

Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.

How a decoder-only LLM is pretrained in practice: data mix, tokens per parameter, batch and LR schedules, bf16, 3D/4D parallelism, checkpoints and loss-spike handling.

Adapting a pretrained LLM with supervised fine-tuning, LoRA/QLoRA adapters and preference optimization (DPO), and evaluating that it improved without forgetting.

Proven setups for training image classifiers and diffusion models: optimizers and LRs, augmentation, label smoothing, EMA, resolution schedules and aspect buckets.

Train a small student to match a large teacher's softened outputs (plus the hard labels) so it keeps most of the teacher's accuracy at a fraction of the cost.

Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.

Keep the k best partial sequences by summed log-probability at every step, set finished ones aside, and return the best (n-best) by a length-normalised score.