mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
AI and ML infrastructure
on-device-inference
on-device inference runtime
llama.cpp, Core ML
System
Techniques · 2
Primitives · 32
on-device-inference
Operator graph partitioning
ml-training · medium: Splitting a model graph across CPU/GPU/NPU backends by supported ops and transfer cost
Graph partitioning
graphs · medium: Split a graph into k balanced parts with few edges between them; multilevel coarsen-partition-refine (METIS, KaHIP) for quality, streaming heuristics (LDG, Fennel) for one-pass scale.
›
Tensor memory planning
ml-training · low: Static buffer reuse from tensor lifetimes so a graph runs in a fixed arena with minimal peak memory
Constrained decoding
agentic-ai · low: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.
›
CPU caches and the memory hierarchy
hardware-os · low: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.
›
mmap and the page cache
hardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.
›
Roofline model
hardware-os · low: Bound a kernel's speed by min(peak FLOP/s, arithmetic intensity × memory bandwidth) to know whether to optimize compute or data movement.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Byte-pair encoding (BPE) tokenizati…
machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.
›
GPU compute shaders
ml-training · low: The vendor-neutral GPU launch model of Metal, Vulkan and WebGPU: a grid of workgroups, shared memory, barriers and subgroups, driven by recorded command buffers.
›
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Model file formats
ml-training · low: Weight file layouts (safetensors, GGUF, ONNX): headers, alignment for mmap, tensor metadata, quantized blocks
Token sampling
ml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.
›
Speculative decoding
ml-training · low: A cheap draft proposes γ tokens, the big model verifies them in one pass, and rejection sampling keeps the target's exact distribution at 2-3x speed.
›
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
Arena allocator
systems · low: Allocate by bumping a pointer through big chunks and free everything at once: a few instructions per allocation for objects sharing a lifetime.
›
Shape inference
compilers · low: Propagating tensor shapes and dtypes through an operator graph, with symbolic dimensions
+ 16 more primitives
System
Techniques · 2
Primitives · 32
on-device-inference
Operator graph partitioning
ml-training · medium: Splitting a model graph across CPU/GPU/NPU backends by supported ops and transfer cost
Graph partitioning
graphs · medium: Split a graph into k balanced parts with few edges between them; multilevel coarsen-partition-refine (METIS, KaHIP) for quality, streaming heuristics (LDG, Fennel) for one-pass scale.
›
Tensor memory planning
ml-training · low: Static buffer reuse from tensor lifetimes so a graph runs in a fixed arena with minimal peak memory
Constrained decoding
agentic-ai · low: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.
›
CPU caches and the memory hierarchy
hardware-os · low: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.
›
mmap and the page cache
hardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.
›
Roofline model
hardware-os · low: Bound a kernel's speed by min(peak FLOP/s, arithmetic intensity × memory bandwidth) to know whether to optimize compute or data movement.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Byte-pair encoding (BPE) tokenizati…
machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.
›
GPU compute shaders
ml-training · low: The vendor-neutral GPU launch model of Metal, Vulkan and WebGPU: a grid of workgroups, shared memory, barriers and subgroups, driven by recorded command buffers.
›
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Model file formats
ml-training · low: Weight file layouts (safetensors, GGUF, ONNX): headers, alignment for mmap, tensor metadata, quantized blocks
Token sampling
ml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.
›
Speculative decoding
ml-training · low: A cheap draft proposes γ tokens, the big model verifies them in one pass, and rejection sampling keeps the target's exact distribution at 2-3x speed.
›
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
Arena allocator
systems · low: Allocate by bumping a pointer through big chunks and free everything at once: a few instructions per allocation for objects sharing a lifetime.
›
Shape inference
compilers · low: Propagating tensor shapes and dtypes through an operator graph, with symbolic dimensions
Topological sort
graphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
BFS and DFS
graphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.
›
Gpu memory coalescing banking
hardware-os · low: Coalesced global loads, shared memory bank conflicts, L2 and HBM bandwidth on GPUs
Virtual memory and the TLB
hardware-os · low: How page tables and TLBs translate every address, what TLB reach means, and when 2 MiB or 1 GiB huge pages pay off on Linux.
›
CUDA and GPU languages
languages · low: Programming GPUs: thousands of lockstep threads, bound by memory bandwidth; libraries first, Triton for fused kernels, CUDA C++ for peak speed.
›
Text normalization
machine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.
›
Cuda graphs
ml-training · low: Capturing a sequence of GPU kernel launches once and replaying it to remove CPU launch overhead
Rejection sampling
ml-training · low: Best-of-n and filtered sampling: generate many candidates, keep those passing a verifier or reward threshold
Transformer block
ml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.
›
Low precision number formats
numerics · low: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scaling
LU and Cholesky factorization
numerics · low: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).
›
Finite state transducer
search-sort · low: Minimal acyclic FST mapping sorted keys to values; compact term dictionaries with prefix/fuzzy search
JSON Schema validation
security · low: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
tensor-runtime
43 %
ml-compiler
22 %
llm-inference
21 %