← Back
Home › Systems › AI and ML infrastructure

data-labeling

data labeling platform

Scale, Label Studio

SystemTechniques · 15Primitives · 53 data-labeling Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Active learningmachine-learning · medium: Choosing which unlabeled examples to label next: uncertainty, margin, diversity and committee samplingImage segmentationmachine-learning · medium: Pixel/instance segmentation networks and promptable segmentation (Mask R-CNN, SAM-style)Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Object detectionmachine-learning · medium: A network turns an image into scored, classified boxes in one pass; anchor-based, anchor-free and set-prediction heads differ in how they assign labels and remove duplicates.›HNSWsearch-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Calibrationmachine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesProcess supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›WebSocketsnetworking · low: Full-duplex message framing over one upgraded HTTP connection; the standard way to push real-time data to browsers.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Annotator agreementstatistics · low: Inter-annotator agreement measures: Cohen and Fleiss kappa, Krippendorff alpha, gold-question accuracyLabel aggregationstatistics · low: Combining noisy labels from several annotators: majority vote, Dawid-Skene EM, annotator weightingOptimistic concurrency controlstorage · low: Run transactions without locks, record what they read, and validate at commit that nothing changed; fastest when conflicts are rare.›Graceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space schedulerRecord replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildPRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Health checkingsystems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›+ 8 more techniques+ 37 more primitives

Similar systems

Share the most building blocks with this one.

issue-tracker 38 %merge-queue 37 %party-lobby 34 %sportsbook 34 %helpdesk 34 %