← Back
Home › Systems › Observability and operations

auto-remediation

runbook automation

Rundeck

SystemTechniques · 14Primitives · 60 auto-remediation DAG workflow executiondistributed · medium: Run a graph of dependent tasks by releasing each one when its inputs are done, longest chain first, and skip any task whose input hashes are unchanged.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Durable workflow executiondistributed · medium: Ordinary code made crash-proof by logging every side-effect result and replaying the code from that log; activities run at least once, workflow logic must be deterministic.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Webhook deliverynetworking · medium: Reliable server-to-server HTTP callbacks: signed POSTs, 2xx-means-done, retries over days with jitter, endpoint disabling, SSRF guards and idempotent receivers.›Coverage-guided fuzzingsecurity · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Tamper-evident audit logsecurity · medium: An append-only log whose Merkle root, signed per batch, lets anyone verify an entry is present and history was never rewritten, with log-size proofs.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Circuit breakersystems · low: Stop calling a failing dependency for a while, probe it with trial calls, and resume on recovery: fail fast instead of piling up timeouts.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesFencing tokensdistributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Process supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Authorization models (RBAC, ABAC, R…security · low: How to decide who may do what to which object; roles for coarse rules, relationships for sharing, and checks on every object.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Secrets managementsecurity · low: Keep keys and passwords out of code and logs, give workloads short-lived identity instead of static keys, and make rotation routine.›SSH protocol (transport, auth, chan…security · low: A three-layer protocol on one TCP connection: encrypted packet transport, user authentication, then multiplexed channels with windows for shells, files and port forwards.›Graceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Token bucketsystems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Content-addressed storagestorage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space scheduler+ 7 more techniques+ 44 more primitives

Similar systems

Share the most building blocks with this one.