← Back
Home › Systems › Data processing and pipelines

workflow-engine

durable workflow orchestrator

Temporal, Airflow

SystemTechniques · 5Primitives · 26 workflow-engine Saga patterndistributed · medium: Split a cross-service operation into local transactions, each paired with an undo; on failure run the undos in reverse, or retry forward.›Transactional outboxdistributed · medium: Write the event as a row in the same local transaction as the state change; a relay publishes it later at-least-once, so state and events never diverge.›Durable job queuedistributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Change data capturestorage · medium: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Incremental sync cursorsdistributed · medium: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Consistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Record replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildEvent sourcingfinance · low: Persist state changes as an append-only stream of immutable events per entity; current state is a fold over them, guarded by expected-version appends.›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Optimistic concurrency controlstorage · low: Run transactions without locks, record what they read, and validate at commit that nothing changed; fastest when conflicts are rare.›Cron schedulingsystems · low: Cron expression parsing and next-fire-time computation, with catch-up and time-zone rulesHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Event deduplication windowdistributed · low: Keep the ids of recently accepted events for a time window and drop repeats; memory stays bounded by rotating time buckets instead of per-entry timers.›Dead-letter queuedistributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Message sequence numbersdistributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Cursor paginationnetworking · low: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tables+ 10 more primitives

Similar systems

Share the most building blocks with this one.

booking 29 %distributed-cron 26 %bpm-low-code 22 %mmo-sharding 22 %llm-workflow-automation 21 %

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

CadenceUber TechnologiesMaestroNetflix