← Back
Home › Systems › AI and ML infrastructure

ml-platform

ML training platform with experiment tracking

MLflow, W&B

SystemTechniques · 11Primitives · 46 ml-platform Feature storemachine-learning · medium: One definition per feature feeds a history store for training and a latest-value store for serving, so training and inference see the same values as of the same moment.›Distributed checkpointingml-training · medium: Sharded, asynchronous checkpoint save/load of model and optimizer state, with resharding to new world sizesExperiment trackingml-training · medium: Logging params, metrics, artifacts and code versions per run with an append-only metric store and comparisonModel registryml-training · medium: Versioned model artifacts with metadata, stages, lineage and immutable content hashesColumnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Dataset versioningstorage · medium: Immutable, content-hashed dataset snapshots with manifests and lineage so experiments are reproducibleWindowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Lttb downsamplingstatistics · low: Largest-Triangle-Three-Buckets: reduce a series to N points while keeping its visual shapeCryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Containers and Kubernetescloud · low: Containers package a service with its dependencies; Kubernetes schedules them across machines. Use the smallest orchestrator that does the job.›GPU cloudcloud · low: Where to rent GPUs for training and inference: hyperscalers for integration, neoclouds for price, spot and marketplaces for cheap batch work; interconnect decides multi-node training.›Gorilla XOR compressioncompression · low: Time-series codec: delta-of-delta timestamps plus XOR of consecutive float bit patterns, storing only changed bits; about 1.4 bytes per point.›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›B+ treestorage · low: Balanced high-fanout search tree with all values in linked leaves; the default on-disk index for point lookups and range scans.›Columnar file formatstorage · low: Row groups, column chunks, pages, encodings and footer stats (Parquet/ORC style) (listed in Moved to the wiki)Atomic file replacementhardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updatesCollective communicationml-training · low: The family of group exchanges (broadcast, reduce, all-gather, reduce-scatter, all-to-all) and the latency-plus-bytes cost model that picks ring, tree or direct algorithms by message size and topology.›Tensor layout transformsml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiencyZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Content-addressed storagestorage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Conditional writesdistributed · low: Put-if-absent / if-match compare-and-swap on objects or keys as a commit primitive+ 4 more techniques+ 30 more primitives

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

ElementWalmartFBLearner FlowMeta PlatformsMichelangeloUber Technologies