mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
CI/CD and developer tooling
container-build
container image builder
BuildKit, Kaniko
System
Techniques · 7
Primitives · 30
container-build
DAG workflow execution
distributed · medium: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.
›
Reproducible builds
security · medium: Bit-identical outputs: fixed timestamps, ordering, paths and toolchains
Incremental computation
systems · medium: Dependency graph of derived values recomputed only when inputs change (reactive cells, spreadsheets)
Layered image build
systems · medium: Build container images as cached, content-addressed layers from a build graph
Lazy image pulling
systems · medium: Start containers before full download by fetching image chunks on demand
Durable job queue
distributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.
›
Hermetic execution
systems · medium: Run actions with only declared inputs in a sandbox and fixed environment
Content-addressed storage
storage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.
›
Tar archive format
storage · low: Ustar/pax tar headers and streaming archives
Overlay filesystem
hardware-os · low: Union mounts (overlayfs) stacking read-only layers under a writable layer
Oci image layout
systems · low: OCI manifests, image index, layer tarballs and config, addressed by digest
Cgroups and namespaces
hardware-os · low: Linux namespaces limit what a process can see; cgroup v2 limits and accounts what it may use. Containers, sandboxes and job schedulers are built from the two.
›
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Deflate
compression · low: DEFLATE/gzip: LZ77 + Huffman, the format behind HTTP gzip, zip and PNG
Zstandard (zstd)
compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.
›
Merkle tree
data-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofs
Topological sort
graphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.
›
Exponential backoff with jitter
systems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.
›
Deterministic serialization
finance · low: Canonical byte encodings (RLP, SSZ, canonical CBOR) so every node hashes identical bytes
Hash consing
data-structures · low: Intern structurally equal immutable terms so equality is pointer comparison and sharing is maximal
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Seekable compressed stream
compression · low: Independently compressed blocks plus an index for random access (BGZF, seekable zstd, eStargz)
Fuse userspace filesystem
hardware-os · low: FUSE protocol: kernel forwards VFS requests to a userspace daemon; caching and splice
+ 14 more primitives
System
Techniques · 7
Primitives · 30
container-build
DAG workflow execution
distributed · medium: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.
›
Reproducible builds
security · medium: Bit-identical outputs: fixed timestamps, ordering, paths and toolchains
Incremental computation
systems · medium: Dependency graph of derived values recomputed only when inputs change (reactive cells, spreadsheets)
Layered image build
systems · medium: Build container images as cached, content-addressed layers from a build graph
Lazy image pulling
systems · medium: Start containers before full download by fetching image chunks on demand
Durable job queue
distributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.
›
Hermetic execution
systems · medium: Run actions with only declared inputs in a sandbox and fixed environment
Content-addressed storage
storage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.
›
Tar archive format
storage · low: Ustar/pax tar headers and streaming archives
Overlay filesystem
hardware-os · low: Union mounts (overlayfs) stacking read-only layers under a writable layer
Oci image layout
systems · low: OCI manifests, image index, layer tarballs and config, addressed by digest
Cgroups and namespaces
hardware-os · low: Linux namespaces limit what a process can see; cgroup v2 limits and accounts what it may use. Containers, sandboxes and job schedulers are built from the two.
›
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Deflate
compression · low: DEFLATE/gzip: LZ77 + Huffman, the format behind HTTP gzip, zip and PNG
Zstandard (zstd)
compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.
›
Merkle tree
data-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofs
Topological sort
graphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.
›
Exponential backoff with jitter
systems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.
›
Deterministic serialization
finance · low: Canonical byte encodings (RLP, SSZ, canonical CBOR) so every node hashes identical bytes
Hash consing
data-structures · low: Intern structurally equal immutable terms so equality is pointer comparison and sharing is maximal
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Seekable compressed stream
compression · low: Independently compressed blocks plus an index for random access (BGZF, seekable zstd, eStargz)
Fuse userspace filesystem
hardware-os · low: FUSE protocol: kernel forwards VFS requests to a userspace daemon; caching and splice
HTTP range requests
networking · low: Ask for byte offsets with Range and get a 206 slice back; the standard way to resume, seek and read big remote files piecewise.
›
Dead-letter queue
distributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.
›
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Idempotency keys
finance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
Merkle dag
data-structures · low: Content-addressed DAG of hashed nodes (git trees/commits) for dedupe and verifiable history
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
BFS and DFS
graphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.
›
Atomic file replacement
hardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updates
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
image-snapshotter
35 %
container-runtime
35 %
build-system
33 %
backup-dr
16 %
remote-build-execution
16 %