← Back
Home › Systems › Data processing and pipelines

web-archive

web archive

Internet Archive

SystemTechniques · 2Primitives · 15 web-archive Url frontiersearch-sort · medium: Prioritised per-host queues with politeness delays for crawlingErasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationBloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Robots exclusion protocolsearch-sort · low: Decide per URL whether a crawler may fetch it from a host's robots.txt (RFC 9309): status-code rules, group choice, longest-match Allow/Disallow, 24 h cache.›URL normalizationsearch-sort · low: Rewrite a URL to one canonical key, in safe RFC 3986 steps plus lossy, site-dependent ones (tracking parameters, query order, fragments), and keep the original beside the key.›Deflatecompression · low: DEFLATE/gzip: LZ77 + Huffman, the format behind HTTP gzip, zip and PNGPrefix compressioncompression · low: Front coding / shared-prefix truncation of sorted keys in blocks and index pagesHtml rewritinggraphics · low: Streaming HTML tokenize-and-rewrite of links/attributes without building a DOMCryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Data scrubbingstorage · low: Background re-read and checksum verification of stored data to catch latent corruptionSstablestorage · low: Sorted string table: data blocks with restart points, index, filter and footerWarc formatstorage · low: WARC records (request, response, metadata) with per-record compression for web archivesd-ary heapdata-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›Token bucketsystems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.›SIMD programminghardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Finite field arithmeticnumerics · low: Prime-field and extension-field arithmetic (Montgomery form, Goldilocks, BabyBear), vectorizedReed-Solomon codessignal-processing · low: Byte-oriented MDS code: n-k parity symbols correct (n-k)/2 unknown symbol errors or n-k known erasures; the workhorse for bursts, QR codes, discs and storage.›

Similar systems

Share the most building blocks with this one.

web-crawler 16 %cold-archive 15 %