← Back
Home › Systems › Sales, marketing and customer service

media-intelligence

media monitoring and social listening

Meltwater, Cision, Brandwatch

SystemTechniques · 22Primitives · 65 media-intelligence Partitioned commit logdistributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Windowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Entity linkingmachine-learning · medium: Map each mention to a knowledge-base id, or to NIL, by generating candidates from an alias table, scoring them on prior and context, and thresholding.›HTML main-text extractionmachine-learning · medium: Decode bytes, tolerate broken markup, drop non-content nodes and score text blocks by length and link density to keep the main run, with headings, metadata and offsets for citation.›Layout analysismachine-learning · medium: Detecting page regions (text blocks, tables, figures) and reading order in documentsNamed entity recognitionmachine-learning · medium: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Optical character recognition (text…machine-learning · medium: Find text lines in an image, read each crop with a CTC or attention model, then correct with a lexicon and score confidence for review routing.›Robots exclusion protocolsearch-sort · low: Decide per URL whether a crawler may fetch it from a host's robots.txt (RFC 9309): status-code rules, group choice, longest-match Allow/Disallow, 24 h cache.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Language identificationmachine-learning · low: Score a text's character n-grams against per-language statistics, after a cheap script check, and return an ISO 639 code with a confidence or "und".›Voice activity detectionmachine-learning · low: Per-frame speech/non-speech decision with smoothing, from an energy threshold over a tracked noise floor up to small neural nets.›Aho-Corasick multi-pattern matchingsearch-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.›Boolean expression indexingsearch-sort · low: Indexing conjunctive targeting expressions so one event finds all matching ads/subscriptions fastSimHashsearch-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Zero-copy I/Onetworking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Text normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›+ 15 more techniques+ 49 more primitives

Similar systems

Share the most building blocks with this one.

influencer-marketing 24 %legal-research 20 %social-media-management 19 %threat-intel-platform 18 %sales-intelligence 18 %