← Back
Home › Systems › Data processing and pipelines

master-data-management

master data management with golden records

Informatica MDM, Reltio, Stibo STEP

SystemTechniques · 16Primitives · 61 master-data-management Human-in-the-loop approvalagentic-ai · medium: Classify each proposed action as auto, ask or deny; park asks as durable pending records a human approves, edits or rejects, executing once, with timeouts that deny.›Address parsinggeometry · medium: Split free-text addresses into components (house, street, city, postcode) with rules or a sequence modelActive learningmachine-learning · medium: Choosing which unlabeled examples to label next: uncertainty, margin, diversity and committee samplingGradient-boosted decision trees (GB…machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Record linkagesearch-sort · medium: Probabilistic record matching (Fellegi-Sunter) with blocking for patient or entity indexesBitemporal datastorage · medium: Valid time plus transaction time on every record for as-of queries, back-dating and correctionsText normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›Union-findgraphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.›Fuzzy string matchingsearch-sort · low: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingClosure tablestorage · low: Storing trees and hierarchies in relational tables (closure table, materialized path) for fast subtree queriesE164 number parsingdata-structures · low: Parse, validate and normalize phone numbers to E.164Edit distancesearch-sort · low: Levenshtein/Damerau distance by DP, bit-parallel Myers variantJSON Schema validationsecurity · low: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.›Entity attribute value schemastorage · low: Schema-flexible EAV / JSON entity storage with typed attribute catalogs and indexesSurvivorship rulesstorage · low: Build the golden record by choosing each attribute from competing source records using trust rank, recency and completeness rulesPublish subscribe event bussystems · low: In-process or on-box topic bus decoupling producers and consumers with bounded queuesStatechartsgraphics · low: State machines with nesting, parallel regions, history and entry/exit actions, so UI and animation logic stays linear in size instead of exploding.›Dual control approvalsecurity · low: Four-eyes / maker-checker approval with quorum policies for sensitive operationsHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Calibrationmachine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›+ 9 more techniques+ 45 more primitives

Similar systems

Share the most building blocks with this one.

crm 18 %music-royalty-collection 18 %identity-graph 18 %aml-monitoring 16 %background-screening 16 %