mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Data processing and pipelines
stream-processor
stateful stream processor
Flink
System
Techniques · 8
Primitives · 32
stream-processor
Chandy lamport snapshots
distributed · medium: Consistent distributed snapshots via markers/barriers (Flink-style checkpoints)
Data shuffle
distributed · medium: Hash/range repartitioning of intermediate data between workers (exchange operator)
Exactly once processing
distributed · medium: Idempotent producers plus transactional offset+output commits for effectively-once results
Windowed stream aggregation
distributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.
›
Two-phase commit
distributed · medium: Atomic commit across shards: participants durably vote yes, then the coordinator logs and broadcasts one decision; blocks if the coordinator is lost.
›
Watermarks
distributed · medium: Event-time progress markers that bound lateness and trigger window results
LSM-tree
storage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.
›
Consumer offsets
distributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.
›
Hierarchical timing wheel
systems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Credit based flow control
networking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)
Zero-copy serialization
networking · low: Wire formats laid out like in-memory structs (FlatBuffers, Cap'n Proto, SBE, rkyv) so readers access fields in place with no decode pass or allocation.
›
Hash join
storage · low: Build a hash table on the smaller input's join key and probe it with every row of the larger one; the linear-time workhorse for equi-joins.
›
Object storage
cloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.
›
Distributed barrier
distributed · low: Rendezvous/barrier for a dynamic worker group: join, agree on membership and rank assignment, restart on change
Copy on write snapshots
storage · low: Point-in-time snapshots by sharing pages/extents and copying only on write
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Range partitioning
distributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.
›
Columnar memory format
storage · low: In-memory column buffers with validity bitmaps and offsets (Arrow style) for zero-copy exchange
Event deduplication window
distributed · low: Keep the ids of recently accepted events for a time window and drop repeats; memory stays bounded by rotating time buckets instead of per-entry timers.
›
Idempotency keys
finance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.
›
Hash aggregation
storage · low: GROUP BY via hash tables with partitioning and spill to disk
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
+ 1 more techniques
+ 16 more primitives
System
Techniques · 8
Primitives · 32
stream-processor
Chandy lamport snapshots
distributed · medium: Consistent distributed snapshots via markers/barriers (Flink-style checkpoints)
Data shuffle
distributed · medium: Hash/range repartitioning of intermediate data between workers (exchange operator)
Exactly once processing
distributed · medium: Idempotent producers plus transactional offset+output commits for effectively-once results
Windowed stream aggregation
distributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.
›
Two-phase commit
distributed · medium: Atomic commit across shards: participants durably vote yes, then the coordinator logs and broadcasts one decision; blocks if the coordinator is lost.
›
Watermarks
distributed · medium: Event-time progress markers that bound lateness and trigger window results
LSM-tree
storage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.
›
External merge sort
search-sort · medium: Sort data larger than RAM: write memory-sized sorted runs, then k-way merge them with large sequential buffers; terabytes in one merge pass.
›
Consumer offsets
distributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.
›
Hierarchical timing wheel
systems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Credit based flow control
networking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)
Zero-copy serialization
networking · low: Wire formats laid out like in-memory structs (FlatBuffers, Cap'n Proto, SBE, rkyv) so readers access fields in place with no decode pass or allocation.
›
Hash join
storage · low: Build a hash table on the smaller input's join key and probe it with every row of the larger one; the linear-time workhorse for equi-joins.
›
Object storage
cloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.
›
Distributed barrier
distributed · low: Rendezvous/barrier for a dynamic worker group: join, agree on membership and rank assignment, restart on change
Copy on write snapshots
storage · low: Point-in-time snapshots by sharing pages/extents and copying only on write
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Range partitioning
distributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.
›
Columnar memory format
storage · low: In-memory column buffers with validity bitmaps and offsets (Arrow style) for zero-copy exchange
Event deduplication window
distributed · low: Keep the ids of recently accepted events for a time window and drop repeats; memory stays bounded by rotating time buckets instead of per-entry timers.
›
Idempotency keys
finance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.
›
Hash aggregation
storage · low: GROUP BY via hash tables with partitioning and spill to disk
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
Buffer pools for zero-allocation I/O
hardware-os · low: Preallocate a fixed set of aligned I/O buffers and recycle them through a free list, so the hot path never mallocs, faults or unmaps. The bounded count doubles as back pressure.
›
K way merge
search-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)
Pattern-defeating quicksort (pdqsor…
search-sort · low: Introsort successor: quicksort with adaptive pivots, linear on sorted and equal-heavy input, heapsort fallback; the unstable sort of Go and (until 1.81) Rust.
›
Memory ordering
concurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.
›
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Message sequence numbers
distributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".
›
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Exponential backoff with jitter
systems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.
›
Slab allocator
systems · low: Pre-carve pages into equal-size slots per size class with an in-place free list: O(1) alloc and free, no external fragmentation, cache-warm reuse.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
feature-store
22 %