← Back
Home › Systems › Observability and operations

alerting

alerting and on-call paging

Alertmanager, PagerDuty

SystemTechniques · 20Primitives · 60 alerting CRDTsdistributed · medium: Data types whose replicas accept writes independently and always converge by merging, with no coordination: counters, sets, maps, registers, text.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Durable job queuedistributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›SWIM gossip membershipdistributed · medium: Each node probes one random peer per period, asks k others on timeout, and piggybacks membership updates; constant load per node at any cluster size.›Webhook deliverynetworking · medium: Reliable server-to-server HTTP callbacks: signed POSTs, 2xx-means-done, retries over days with jitter, endpoint disabling, SSRF guards and idempotent receivers.›Coverage-guided fuzzingsecurity · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Hierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Calendar date arithmeticsystems · low: Civil dates, time zones (tzdb), DST gaps, month-end anchoring and duration mathEvent deduplication windowdistributed · low: Keep the ids of recently accepted events for a time window and drop repeats; memory stays bounded by rotating time buckets instead of per-entry timers.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesProcess supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Graceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Fractional indexingdistributed · low: Give every list item a sortable string key and generate a new key strictly between its neighbours, so a move or insert writes one row.›Lamport clocksdistributed · low: One integer per process, max-plus-one on receive; gives a total order consistent with causality using 8 bytes per timestamp.›Vector clocksdistributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space schedulerRecord replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildPRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Dead-letter queuedistributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›+ 13 more techniques+ 44 more primitives

Similar systems

Share the most building blocks with this one.