← Back
Home › Systems › Observability and operations

chaos-engineering

fault injection

Chaos Monkey

SystemTechniques · 19Primitives · 60 chaos-engineering Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Fault injectiondistributed · medium: Inject latency, errors, kills and partitions via proxies, netem or hooksPrimary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Reverse proxynetworking · medium: A server-side intermediary that terminates client connections and forwards each request to one of a pool of upstream servers, with routing, health checks, timeouts and safe retries.›Coverage-guided fuzzingsecurity · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Syscall interceptionsecurity · medium: Intercept syscalls via ptrace, seccomp user notify or a user-space kernelHypothesis testingstatistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Health checkingsystems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Ebpfhardware-os · low: Verified in-kernel programs on hooks (kprobes, XDP, tc) with mapsHdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesCgroups and namespaceshardware-os · low: Linux namespaces limit what a process can see; cgroup v2 limits and accounts what it may use. Containers, sandboxes and job schedulers are built from the two.›Process supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Circuit breakersystems · low: Stop calling a failing dependency for a while, probe it with trial calls, and resume on recovery: fail fast instead of piling up timeouts.›Graceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space schedulerRecord replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildPRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Transparent interceptionnetworking · low: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changesLeasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›HTTP/1.1 message parsingnetworking · low: Parse HTTP/1.1 requests and responses incrementally from a byte stream: RFC 9112 body-length rules, chunked decoding and the strictness that stops request smuggling.›HTTP/2networking · low: Binary multiplexed framing for HTTP over one TCP connection: streams, per-stream and per-connection credit windows, HPACK header compression and the limits that contain abuse.›+ 12 more techniques+ 44 more primitives

Similar systems

Share the most building blocks with this one.