← Back
Home › Systems › AI and ML infrastructure

pretraining-data

web-scale pretraining data pipeline

FineWeb, Common Crawl

SystemTechniques · 13Primitives · 49 pretraining-data Data loading pipelinesmachine-learning · medium: Turn files on disk into shuffled, sharded, resumable GPU batches fast enough that the accelerator never waits.›HTML main-text extractionmachine-learning · medium: Decode bytes, tolerate broken markup, drop non-content nodes and score text blocks by length and link density to keep the main run, with headings, metadata and offsets for citation.›Named entity recognitionmachine-learning · medium: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Text classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Data mixture weightingml-training · medium: Choosing sampling weights across data sources/domains (temperature sampling, DoReMi-style proxies, epoch limits)External merge sortsearch-sort · medium: Sort data larger than RAM: write memory-sized sorted runs, then k-way merge them with large sequential buffers; terabytes in one merge pass.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Byte-pair encoding (BPE) tokenizati…machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Zstandard (zstd)compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.›Language identificationmachine-learning · low: Score a text's character n-grams against per-language statistics, after a cheap script check, and return an ISO 639 code with a confidence or "und".›Ngram language modelmachine-learning · low: N-gram counts with Kneser-Ney smoothing, compact storage (KenLM-style) and perplexitySuffix arraysearch-sort · low: Sorted array of all suffix start positions of a text; finds every occurrence of any substring by binary search, at 4-5 bytes per text byte.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›mmap and the page cachehardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Sequence packingml-training · low: Packing variable-length sequences into fixed-size batches with attention masks to avoid padding wastePRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Text normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›Transformer blockml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Convolutional codes and the Viterbi…signal-processing · low: Streaming error-correcting code: each output bit is a parity of the last K input bits; the Viterbi decoder finds the most likely path through a 2^(K-1)-state trellis.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Rolling hashdata-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matching+ 6 more techniques+ 33 more primitives

Similar systems

Share the most building blocks with this one.

threat-intel-platform 18 %web-search 18 %logging 17 %dlp 17 %ediscovery 15 %