← Back
Home › Systems › AI and ML infrastructure

embedding-service

embedding service at scale

OpenAI embeddings

SystemTechniques · 4Primitives · 35 embedding-service Contrastive learningmachine-learning · medium: Train an encoder so two views of the same thing score higher than every other pair in the batch; InfoNCE with a temperature, no labels needed.›Dynamic batchingml-training · medium: Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.›HNSWsearch-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.›IVF-PQsearch-sort · medium: Inverted file plus product quantization: cluster vectors into lists, store each as a few bytes of codes, scan only the nearest lists via table lookups.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›SIMD programminghardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Byte-pair encoding (BPE) tokenizati…machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.›Matryoshka embeddingsmachine-learning · low: Training embeddings whose prefixes are usable at lower dimensions for cheaper searchFlashAttentionml-training · low: Exact attention computed tile by tile in on-chip SRAM with online softmax, so the N×N score matrix never touches HBM; 2-4x faster, O(N) memory.›Sequence packingml-training · low: Packing variable-length sequences into fixed-size batches with attention masks to avoid padding wasteTransformer blockml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Weight quantizationml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.›LRU and CLOCKsystems · low: LRU evicts the least recently used entry via hash map plus linked list; CLOCK approximates it with a reference bit, so hits take no lock.›Data augmentationmachine-learning · low: Generate label-preserving variants of each training example on the fly to enlarge the effective dataset and bake in invariances.›Loss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Negative samplingmachine-learning · low: Train by contrasting each observed positive with a few sampled negatives instead of normalising over every candidate; the choice of sampler decides bias and quality.›MPMC bounded queueconcurrency · low: Vyukov's array queue for many producers and consumers: each slot carries a sequence number, so one CAS per operation claims a slot without locks.›Backpressuresystems · low: Bound every queue and make a full queue push back on the producer by blocking, dropping, rejecting or withholding credit.›Hierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›d-ary heapdata-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›+ 19 more primitives

Similar systems

Share the most building blocks with this one.

visual-search 28 %machine-translation 21 %image-generation 20 %recommender 19 %prompt-injection-detection 18 %