← Back
Home › Systems › Data processing and pipelines

sql-transform

SQL transformation layer

dbt

SystemTechniques · 4Primitives · 18 sql-transform Column level lineagestorage · medium: Derive which source columns feed each output column by analysing SQL plansIncremental view maintenancestorage · medium: Update materialized views from deltas instead of recomputingChange data capturestorage · medium: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Incremental sync cursorsdistributed · medium: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Recursive-descent parsingcompilers · low: Hand-written top-down parser with one function per grammar rule; simple, fast, with full control over error messages and recovery.›Topological sortgraphs · low: Order the vertices of a DAG so every edge points forward; the core of build systems, schedulers and dependency resolution, with cycle detection for free.›Slowly changing dimensionsstorage · low: SCD type 1/2 history tables with validity rangesUpsert mergestorage · low: MERGE/upsert by key with deduplication and late-arriving data rulesName resolutioncompilers · low: Bind every use of a name to its declaration with a scope stack, handling shadowing, hoisting and closure capture in one AST walk.›BFS and DFSgraphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.›Relational algebrastorage · low: Relational operators and equivalence rules: selection pushdown, join reordering, projectionsHash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskConsumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Message sequence numbersdistributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Cursor paginationnetworking · low: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesFencing tokensdistributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›CRC32C checksumshardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›fsync, O_DSYNC and O_DIRECThardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›+ 2 more primitives

Similar systems

Share the most building blocks with this one.

data-catalog 18 %