← Back
Home › Systems › Data processing and pipelines

feature-store

ML feature store

Feast

SystemTechniques · 7Primitives · 20 feature-store Windowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Watermarksdistributed · medium: Event-time progress markers that bound lateness and trigger window resultsChange data capturestorage · medium: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Incremental view maintenancestorage · medium: Update materialized views from deltas instead of recomputingLSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Point in time joinstorage · medium: As-of join that picks the latest feature value at or before each event timestampIncremental sync cursorsdistributed · medium: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Consistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Chunked backfillstorage · low: Process old data in bounded key ranges, one short transaction per chunk, with a saved checkpoint and idempotent writes, so a crash resumes and live traffic never waits.›Columnar file formatstorage · low: Row groups, column chunks, pages, encodings and footer stats (Parquet/ORC style) (listed in Moved to the wiki)Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›d-ary heapdata-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Message sequence numbersdistributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Relational algebrastorage · low: Relational operators and equivalence rules: selection pushdown, join reordering, projectionsSort-merge joinstorage · low: Sort both inputs on the key, then advance two cursors in lockstep and emit matching runs; cheap on sorted inputs and good for range joins.›Time partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›Window functionsstorage · low: ROW_NUMBER, rank, running and frame aggregates via sort and segment treesCursor paginationnetworking · low: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesFencing tokensdistributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›+ 4 more primitives

Similar systems

Share the most building blocks with this one.

stream-processor 22 %