← Back
Home › Systems › Simulation and scientific computing

nowcasting

radar precipitation nowcasting

DeepMind DGMR, RainViewer

SystemTechniques · 8Primitives · 54 nowcasting DAG workflow executiondistributed · medium: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.›Partitioned commit logdistributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Data loading pipelinesmachine-learning · medium: Turn files on disk into shuffled, sharded, resumable GPU batches fast enough that the accelerator never waits.›Motion estimationmedia · medium: Block matching and sub-pixel motion search for inter predictionSemi lagrangian advectionphysics · medium: Trace characteristics back and interpolate; unconditionally stable large time stepsChunked array storagestorage · medium: Cut an N-dimensional array into a regular grid of independently compressed blocks, so a slice read touches only the overlapping blocks and writers to different blocks run in parallel.›Durable job queuedistributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Map projectionsgeometry · low: Web Mercator, UTM and equal-area projections, datums and coordinate transformsMap tile pyramidgeometry · low: Z/x/y tile pyramids, quadkeys and overviews for multi-resolution maps and imageryImage warping remapgraphics · low: Resampling with lookup maps: resize, lens undistortion, homographies, bilinear/bicubicEvaluation metricsmachine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Recurrent networksmachine-learning · low: GRU and LSTM cells for streaming sequence modelsConvolutional networksml-training · low: Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.›Diffusion modelsml-training · low: Train a network to undo Gaussian noise at every noise level, then generate by integrating from pure noise to data; the standard for images, video and audio.›Mixed-precision trainingml-training · low: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.›Common alerting protocolnetworking · low: CAP alert message format: areas as polygons/circles, severity, urgency, multilingual info blocksFast Fourier transformnumerics · low: Computes the DFT in O(n log n) by recursively splitting into even and odd halves; the basis of fast convolution and spectral methods.›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Content-addressed storagestorage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›+ 1 more techniques+ 38 more primitives

Similar systems

Share the most building blocks with this one.

flood-forecasting 24 %earth-observation 21 %ml-weather-model 16 %