mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
AI and ML infrastructure
tensor-runtime
portable tensor runtime
ONNX Runtime
System
Techniques · 4
Primitives · 26
tensor-runtime
Dataflow analysis
compilers · medium: Compute a fact per program point (live variables, reaching definitions, available expressions) by iterating transfer functions over the CFG to a fixed point.
›
Work-stealing deque
concurrency · medium: Per-worker deque: the owner pushes and pops at the bottom without contention, idle workers steal from the top; the core of Rayon, Tokio and Go.
›
Operator graph partitioning
ml-training · medium: Splitting a model graph across CPU/GPU/NPU backends by supported ops and transfer cost
Graph partitioning
graphs · medium: Split a graph into k balanced parts with few edges between them; multilevel coarsen-partition-refine (METIS, KaHIP) for quality, streaming heuristics (LDG, Fennel) for one-pass scale.
›
Topological sort
graphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.
›
Shape inference
compilers · low: Propagating tensor shapes and dtypes through an operator graph, with symbolic dimensions
Tensor memory planning
ml-training · low: Static buffer reuse from tensor lifetimes so a graph runs in a fixed arena with minimal peak memory
mmap and the page cache
hardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Cuda graphs
ml-training · low: Capturing a sequence of GPU kernel launches once and replaying it to remove CPU launch overhead
GPU compute shaders
ml-training · low: The vendor-neutral GPU launch model of Metal, Vulkan and WebGPU: a grid of workgroups, shared memory, barriers and subgroups, driven by recorded command buffers.
›
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Model file formats
ml-training · low: Weight file layouts (safetensors, GGUF, ONNX): headers, alignment for mmap, tensor metadata, quantized blocks
Tensor layout transforms
ml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiency
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
Protobuf wire format
networking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolution
Arena allocator
systems · low: Allocate by bumping a pointer through big chunks and free everything at once: a few instructions per allocation for objects sharing a lifetime.
›
Control-flow graph
compilers · low: Split a linear instruction list into basic blocks and connect them by jumps; the graph every optimizer, SSA builder and dataflow pass walks.
›
Epoch-based reclamation
concurrency · low: Threads announce a global epoch while touching shared lock-free data; garbage retired in epoch e is freed once the epoch reaches e + 2.
›
+ 10 more primitives
System
Techniques · 4
Primitives · 26
tensor-runtime
Dataflow analysis
compilers · medium: Compute a fact per program point (live variables, reaching definitions, available expressions) by iterating transfer functions over the CFG to a fixed point.
›
Work-stealing deque
concurrency · medium: Per-worker deque: the owner pushes and pops at the bottom without contention, idle workers steal from the top; the core of Rayon, Tokio and Go.
›
Operator graph partitioning
ml-training · medium: Splitting a model graph across CPU/GPU/NPU backends by supported ops and transfer cost
Graph partitioning
graphs · medium: Split a graph into k balanced parts with few edges between them; multilevel coarsen-partition-refine (METIS, KaHIP) for quality, streaming heuristics (LDG, Fennel) for one-pass scale.
›
Topological sort
graphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.
›
Shape inference
compilers · low: Propagating tensor shapes and dtypes through an operator graph, with symbolic dimensions
Tensor memory planning
ml-training · low: Static buffer reuse from tensor lifetimes so a graph runs in a fixed arena with minimal peak memory
mmap and the page cache
hardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Cuda graphs
ml-training · low: Capturing a sequence of GPU kernel launches once and replaying it to remove CPU launch overhead
GPU compute shaders
ml-training · low: The vendor-neutral GPU launch model of Metal, Vulkan and WebGPU: a grid of workgroups, shared memory, barriers and subgroups, driven by recorded command buffers.
›
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Model file formats
ml-training · low: Weight file layouts (safetensors, GGUF, ONNX): headers, alignment for mmap, tensor metadata, quantized blocks
Tensor layout transforms
ml-training · low: Choosing and converting tensor layouts (NCHW/NHWC, blocked, tiled) for kernel efficiency
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
Protobuf wire format
networking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolution
Arena allocator
systems · low: Allocate by bumping a pointer through big chunks and free everything at once: a few instructions per allocation for objects sharing a lifetime.
›
Control-flow graph
compilers · low: Split a linear instruction list into basic blocks and connect them by jumps; the graph every optimizer, SSA builder and dataflow pass walks.
›
Epoch-based reclamation
concurrency · low: Threads announce a global epoch while touching shared lock-free data; garbage retired in epoch e is freed once the epoch reaches e + 2.
›
Memory ordering
concurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
BFS and DFS
graphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.
›
Gpu memory coalescing banking
hardware-os · low: Coalesced global loads, shared memory bank conflicts, L2 and HBM bandwidth on GPUs
Virtual memory and the TLB
hardware-os · low: How page tables and TLBs translate every address, what TLB reach means, and when 2 MiB or 1 GiB huge pages pay off on Linux.
›
CUDA and GPU languages
languages · low: Programming GPUs: thousands of lockstep threads, bound by memory bandwidth; libraries first, Triton for fused kernels, CUDA C++ for peak speed.
›
Low precision number formats
numerics · low: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scaling
LU and Cholesky factorization
numerics · low: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).
›
Copy for AI
Similar systems
Share the most building blocks with this one.
on-device-inference
43 %
ml-compiler
31 %
autodiff-framework
21 %
array-computing
17 %
incremental-compiler
15 %