← Back
Home › Systems › Databases and storage engines

online-schema-migration

online schema migration without downtime

gh-ost, pg_repack

SystemTechniques · 9Primitives · 24 online-schema-migration Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Change data capturestorage · medium: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Expand contract migrationstorage · medium: Multi-step schema changes (expand, migrate, contract) that keep old and new code workingMVCCstorage · medium: Keep timestamped versions of each row so readers see a consistent snapshot without blocking writers; the default concurrency control in modern databases.›Online schema changestorage · medium: Shadow table, backfill, change replay and atomic cutover; expand-contract migrationsAdaptive concurrency limitssystems · medium: AIMD/gradient limits on in-flight requests driven by observed latency, for backpressure and overload protectionIncremental sync cursorsdistributed · medium: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Chunked backfillstorage · low: Process old data in bounded key ranges, one short transaction per chunk, with a saved checkpoint and idempotent writes, so a crash resumes and live traffic never waits.›Merkle treedata-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofsTwo-phase lockingstorage · low: Transactions take shared/exclusive locks and release none until they stop acquiring; strict 2PL holds them to commit and guarantees serializability.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Health checkingsystems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Message sequence numbersdistributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Counting semaphoreconcurrency · low: Counting and binary semaphores for task signalling and resource countingHdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesSliding-window counterssystems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.›Cursor paginationnetworking · low: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesTopological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Cryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Deterministic bucketingsystems · low: Hash a stable unit id with a per-purpose salt into a bucket; "in" means bucket below the percentage, so assignment is stateless, repeatable and only grows on ramp-up.›+ 2 more techniques+ 8 more primitives

Similar systems

Share the most building blocks with this one.

migration-ci 25 %