← Back
Home › Systems › Security and identity

threat-intel-platform

threat intelligence collection and fusion: STIX/TAXII, dark-web crawling, NLP IOC extraction, scoring and ageing, graph, sharing

Recorded Future, MISP, Google Threat Intelligence

SystemTechniques · 16Primitives · 51 threat-intel-platform Graph community detectiongraphs · medium: Louvain/label propagation community detection for rings, fraud clusters and campaignsPagerankgraphs · medium: Power-iteration PageRank on a sparse graph with damping and dangling nodesHTML main-text extractionmachine-learning · medium: Decode bytes, tolerate broken markup, drop non-content nodes and score text blocks by length and link density to keep the main run, with headings, metadata and offsets for citation.›Knowledge graph embeddingsmachine-learning · medium: Embedding entities and relations (TransE, RotatE, ComplEx) for link predictionNamed entity recognitionmachine-learning · medium: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Relation extractionmachine-learning · medium: Extracting (subject, relation, object) triples from text with classifiers or LLMsText classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Time decay scoringsearch-sort · low: Scoring items by recency with exponential decay or half-life, combined with relevanceBM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Dns blocklistsnetworking · low: DNSBL/URIBL reputation lookups via reversed-IP DNS queriesRdap whoisnetworking · low: Look up domain and IP registration data via RDAP (JSON over HTTP, IANA bootstrap) or WHOIS text protocol on port 43SimHashsearch-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.›Url categorizationsearch-sort · low: Classify URLs and domains into categories using lists, patterns and classifiersFuzzy hashingsecurity · low: Similarity-preserving file hashes (ssdeep CTPH, TLSH) to find variants of known filesStix taxii exchangesecurity · low: STIX object model and TAXII feeds for sharing indicators with markings and sightingsPublish subscribe event bussystems · low: In-process or on-box topic bus decoupling producers and consumers with bounded queuesUnion-findgraphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.›PRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Sparse matrices (CSR)numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›Reproducible reductionnumerics · low: Order-independent floating-point sums (fixed order, Kahan, binned) for bit-for-bit reproducibilityText normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›+ 9 more techniques+ 35 more primitives

Similar systems

Share the most building blocks with this one.

news-ranking 26 %media-intelligence 18 %knowledge-graph 18 %pretraining-data 18 %sales-intelligence 17 %