← Back
Home › Systems › AI applications

visual-search

AI applications | visual-search — search with the camera or a screen region: detect and segment the object, embed it, retrieve products, places, plants, text and entities from a billion-scale index, rank results and answer with a multimodal model; partly on device

Google Lens and Circle to Search, Apple Visual Intelligence and Visual Look Up, Pinterest Lens, Bing Visual Search, Amazon Lens

SystemTechniques · 15Primitives · 44 visual-search Scatter-gather querydistributed · medium: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.›Contrastive learningmachine-learning · medium: Train an encoder so two views of the same thing score higher than every other pair in the batch; InfoNCE with a temperature, no labels needed.›Entity linkingmachine-learning · medium: Map each mention to a knowledge-base id, or to NIL, by generating candidates from an alias table, scoring them on prior and context, and thresholding.›Image segmentationmachine-learning · medium: Pixel/instance segmentation networks and promptable segmentation (Mask R-CNN, SAM-style)Learning to rankmachine-learning · medium: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.›Object detectionmachine-learning · medium: A network turns an image into scored, classified boxes in one pass; anchor-based, anchor-free and set-prediction heads differ in how they assign labels and remove duplicates.›Optical character recognition (text…machine-learning · medium: Find text lines in an image, read each crop with a CTC or attention model, then correct with a lexicon and score confidence for review routing.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Transformer blockml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Image resampling filtersgraphics · low: Resize an image by convolving with a kernel (box, triangle, cubic, Lanczos) whose width scales with the shrink factor, done as two separable 1-D passes with precomputed weights.›Matryoshka embeddingsmachine-learning · low: Training embeddings whose prefixes are usable at lower dimensions for cheaper searchWeight quantizationml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.›Perceptual hashingsearch-sort · low: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.›Query expansionsearch-sort · low: Rewriting queries with synonyms, pseudo-relevance feedback or LLM-generated variants (multi-query, HyDE)K way mergesearch-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)Top-k selectionsearch-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.›Deadline propagationsystems · low: Carry an absolute deadline and cancellation along a call chainHedged requestssystems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latencyData augmentationmachine-learning · low: Generate label-preserving variants of each training example on the fly to enlarge the effective dataset and bake in invariances.›Loss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Negative samplingmachine-learning · low: Train by contrasting each observed positive with a few sampled negatives instead of normalising over every candidate; the choice of sampler decides bias and quality.›Non-maximum suppressionmachine-learning · low: Keep the best-scoring candidate in each neighbourhood and drop the rest; greedy NMS, Soft-NMS and 1-D peak picking share one loop.›Convolutional networksml-training · low: Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.›+ 8 more techniques+ 28 more primitives

Similar systems

Share the most building blocks with this one.

embedding-service 28 %rag 21 %recommender 15 %