← Back
Home › Systems › Data processing and pipelines

task-queue

message queue and background job queue

RabbitMQ, SQS

SystemTechniques · 2Primitives · 25 task-queue Raftdistributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.›State-machine snapshotsdistributed · medium: Periodically save the state a log has produced at index i so the log prefix can be deleted; recovery and catch-up become load snapshot plus replay suffix.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›MPMC bounded queueconcurrency · low: Vyukov's array queue for many producers and consumers: each slot carries a sequence number, so one CAS per operation claims a slot without locks.›d-ary heapdata-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›Topic matching triedata-structures · low: Trie over topic levels with + and # wildcards for subscription matchingDead-letter queuedistributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Credit based flow controlnetworking · low: Receiver-granted credits/windows bound in-flight data (HTTP/2 windows, AMQP prefetch, backpressure)Fair queuingnetworking · low: Deficit round robin / weighted fair queuing to share capacity among tenants or flowsGroup commitstorage · low: Let concurrent committers share one log write and fsync; throughput grows with concurrency while each commit still waits at most about two fsyncs.›Hierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Copy-on-write forkhardware-os · low: fork() gives the child the parent's whole address space by sharing pages read-only and copying one only when someone writes it; vfork and posix_spawn skip even that.›Copy on write snapshotsstorage · low: Point-in-time snapshots by sharing pages/extents and copying only on writeFalse sharingconcurrency · low: Independent variables written by different cores on one cache line force the line to bounce between cores; pad or regroup hot data to fix it.›Futex-based mutexconcurrency · low: A mutex that is one atomic word in user space and enters the kernel (futex wait/wake) only under contention; how std and glibc locks work.›+ 9 more primitives

Similar systems

Share the most building blocks with this one.

distributed-cron 32 %workflow-engine 19 %realtime-payments 18 %coordination-store 17 %telecom-charging 16 %