← Back
Home › Systems › Simulation and scientific computing

ml-weather-model

ML weather forecasting

GraphCast, ECMWF AIFS

SystemTechniques · 5Primitives · 45 ml-weather-model Data loading pipelinesmachine-learning · medium: Turn files on disk into shuffled, sharded, resumable GPU batches fast enough that the accelerator never waits.›Graph neural networksmachine-learning · medium: Message-passing neural networks over graphs (GCN, GAT, equivariant) for molecules and relationsConservative regriddingnumerics · medium: Remap fields between grids preserving integrals, using overlap areas (first and second order)Chunked array storagestorage · medium: Cut an N-dimensional array into a regular grid of independently compressed blocks, so a slice read touches only the overlapping blocks and writers to different blocks run in parallel.›Polygon clippinggeometry · medium: Polygon intersection/union/difference (Weiler-Atherton, Martinez) for spatial predicatesAttention variantsmachine-learning · low: Multi-head, multi-query, grouped-query, sliding-window and latent attention; same softmax(QKᵀ/√d)V core, different trade-offs between quality and KV-cache size.›Normalization layersmachine-learning · low: Standardize activations per batch, token or group with a learned scale so deep nets train fast and stably; RMSNorm in LLMs, BatchNorm in convnets.›Spherical gridsnumerics · low: Lat-lon, cubed-sphere and icosahedral grids for global models; pole and area issuesHealpix sky pixelizationgeometry · low: HEALPix and HTM equal-area hierarchical sphere pixelization for sky indexing and cross-matchingEvaluation metricsmachine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Loss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›AdamWml-training · low: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.›Diffusion modelsml-training · low: Train a network to undo Gaussian noise at every noise level, then generate by integrating from pure noise to data; the standard for images, video and audio.›Diffusion samplersml-training · low: DDPM/DDIM and ODE solvers (DPM-Solver, Euler, Heun) for few-step sampling from diffusion and flow modelsFlashAttentionml-training · low: Exact attention computed tile by tile in on-chip SRAM with online softmax, so the N×N score matrix never touches HBM; 2-4x faster, O(N) memory.›Gradient checkpointingml-training · low: Save only some activations in the forward pass and recompute the rest during backward, trading about 33% extra compute for O(√n) activation memory.›Learning-rate warmup and cosine dec…ml-training · low: Ramp the learning rate up linearly for the first ~1% of steps, then decay it along a half cosine to ~10% of peak.›Mixed-precision trainingml-training · low: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.›Tensor parallelismml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.›Transformer blockml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›ZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›+ 29 more primitives

Similar systems

Share the most building blocks with this one.

llm-pretraining 34 %surrogate-modeling 30 %protein-structure 30 %robot-policy 25 %world-model 24 %