HTML main-text extraction

Deep dive

How it works

Parts:

• decoder — bytes to text: BOM, HTTP charset, <meta> prescan, detector
• tokenizer / tree builder — tags and text with error recovery (HTML parsing)
• node filter — drops script, style, nav, footer, aside, form, hidden subtrees
• block segmenter — block-level tags end a block; each keeps tag path, text, link-text length
• scorer — per-block features (words, link density, tag) to keep or drop
• assembler — main run of kept blocks, heading path, offsets, metadata

Operations:

1. Decode. WHATWG order: byte order mark, transport charset (HTTP header), <meta charset> prescan of the first 1024 bytes (tentative), a detector, a locale default.
2. Parse. Unclosed <p> and stray </div> are normal; an HTML5 tree builder matches the browser, a tolerant tokenizer suffices for text.
3. Drop. Remove those subtrees, hidden / display:none elements, comments, script/style text.
4. Segment. Close a block at each block-level tag. Collapse whitespace, decode entities, count link_chars (text inside <a>).
5. Score. Label blocks by own and neighbour features. jusText defaults: link density above 0.2 is boilerplate; under 70 characters short, over 200 good, between by stop-word share (0.30-0.32).
6. Or score containers. Readability scores candidate containers by their paragraph text, propagates to parents, merges siblings; charThreshold defaults to 500 characters.
7. Assemble. Keep the longest run of good blocks plus the headings before them; join with blank lines; track heading stack and byte offsets.
8. Metadata. <title>, OpenGraph, canonical link and JSON-LD (Article, Product offers.price) give title, author, date, price.

bytes → decode → parse → drop nodes → blocks (path, text, link_chars) → score
 nav ✗ │ ad-links ✗ │ h1 ✓ p ✓ p ✓ p ✓ │ related-links ✗ │ footer ✗
                      └─ main run ─► text + heading path + offsets

Code

const VOID: [&str; 6] = ["br", "hr", "img", "input", "meta", "link"];
const DROP: [&str; 7] = ["script", "style", "nav", "footer", "aside", "form", "noscript"];
const BLOCK: [&str; 19] = ["p", "div", "h1", "h2", "h3", "h4", "h5", "h6", "li", "ul", "ol", "tr", "td",
    "table", "article", "section", "header", "blockquote", "br"];

#[derive(Debug, Default, Clone)]
pub struct Block { pub path: String, pub text: String, pub link_chars: usize, pub words: usize }

impl Block {
    fn link_density(&self) -> f32 { self.link_chars as f32 / self.text.len().max(1) as f32 }
    fn heading(&self) -> bool { self.path.rsplit('/').next().map_or(false, |t| t.len() == 2 && t.starts_with('h')) }
}

fn entity(s: &str) -> String {
    s.replace("&nbsp;", " ").replace("&lt;", "<").replace("&gt;", ">").replace("&quot;", "\"").replace("&amp;", "&")
}

/// Tolerant tokenizer: never fails; unknown/unclosed markup just ends up as text or is ignored.
pub fn blocks(html: &str) -> Vec<Block> {
    let b = html.as_bytes();
    let (mut out, mut cur) = (Vec::new(), Block::default());
    let mut stack: Vec<(String, bool)> = Vec::new(); // (tag, dropped); drop is inherited from ancestors
    let (mut i, mut link_depth) = (0, 0usize);
    let flush = |cur: &mut Block, out: &mut Vec<Block>, stack: &Vec<(String, bool)>| {
        let t = cur.text.split_whitespace().collect::<Vec<_>>().join(" ");
        if !t.is_empty() { cur.words = t.split(' ').count(); cur.text = entity(&t); out.push(cur.clone()); }
        *cur = Block::default();
        cur.path = stack.iter().map(|(n, _)| n.as_str()).collect::<Vec<_>>().join("/");
    };
    while i < b.len() {
        if b[i] != b'<' {
            let j = html[i..].find('<').map_or(b.len(), |p| i + p);
            if !stack.last().map_or(false, |s| s.1) {
                cur.text.push_str(&html[i..j]);
                if link_depth > 0 { cur.link_chars += html[i..j].trim().len(); }
            }
            i = j; continue;
        }
        if html[i..].starts_with("<!--") { i = html[i..].find("-->").map_or(b.len(), |p| i + p + 3); continue; }
        // end of tag: first '>' outside quotes
        let (mut j, mut q) = (i + 1, 0u8);
        while j < b.len() && (q != 0 || b[j] != b'>') {
            if q == 0 && (b[j] == b'"' || b[j] == b'\'') { q = b[j] } else if b[j] == q { q = 0 }
            j += 1;
        }
        let inner = &html[i + 1..j.min(b.len())];
        i = j + 1;
        let closing = inner.starts_with('/');
        let body = inner.trim_start_matches('/');
        let name: String = body.chars().take_while(|c| c.is_ascii_alphanumeric()).collect::<String>().to_ascii_lowercase();
        if name.is_empty() { continue; } // <!doctype>, stray '<'
        let block = BLOCK.contains(&name.as_str());
        if closing {
            if let Some(p) = stack.iter().rposition(|s| s.0 == name) { // pop to the match; unmatched closers are ignored
                if block { flush(&mut cur, &mut out, &stack); }
                stack.truncate(p);
                if block { cur.path = stack.iter().map(|(n, _)| n.as_str()).collect::<Vec<_>>().join("/"); }
                if name == "a" && link_depth > 0 { link_depth -= 1; }
            }
            continue;
        }
        let parent_drop = stack.last().map_or(false, |s| s.1);
        let hidden = body.split_whitespace().any(|t| t == "hidden" || t.starts_with("hidden="))
            || body.replace(' ', "").contains("display:none");
        if block { flush(&mut cur, &mut out, &stack); }
        if VOID.contains(&name.as_str()) { continue; }
        if (name == "script" || name == "style") && !inner.ends_with('/') { // raw text: skip to the end tag
            let pat = format!("</{name}").into_bytes();
            i = (i..b.len()).find(|&k| b[k..].len() >= pat.len() && b[k..k + pat.len()].eq_ignore_ascii_case(&pat)).unwrap_or(b.len());
            continue;
        }
        stack.push((name.clone(), parent_drop || hidden || DROP.contains(&name.as_str())));
        if name == "a" { link_depth += 1; }
        if block { cur.path = stack.iter().map(|(n, _)| n.as_str()).collect::<Vec<_>>().join("/"); }
    }
    flush(&mut cur, &mut out, &stack);
    out
}

/// Boilerpipe/jusText-style labelling; the thresholds are illustrative, tune on your own pages.
pub fn main_text(html: &str) -> String {
    let bs = blocks(html);
    let good = |b: &Block| b.words >= 10 && b.link_density() < 0.33;
    let keep: Vec<bool> = (0..bs.len()).map(|k| {
        let (prev, next) = (k > 0 && good(&bs[k - 1]), k + 1 < bs.len() && good(&bs[k + 1]));
        good(&bs[k])
            || (bs[k].heading() && next && bs[k].link_density() < 0.5)                  // title of a good block
            || (bs[k].words >= 3 && bs[k].link_density() == 0.0 && prev && next)       // short block inside the body
    }).collect();
    // main = the run of consecutive kept blocks with the most words
    let (mut best, mut cur, mut start) = ((0, 0, 0), 0, 0);
    for k in 0..=bs.len() {
        if k < bs.len() && keep[k] { if cur == 0 { start = k; } cur += bs[k].words; }
        else { if cur > best.0 { best = (cur, start, k); } cur = 0; }
    }
    bs[best.1..best.2].iter().map(|b| b.text.as_str()).collect::<Vec<_>>().join("\n\n")
}

fn main() {
    let page = r#"<script>var s = "<p>no</p>";</script><nav><a href="/">Home</a> <a href="/n">News</a></nav>
<div>Sponsored: <a href="/1">Ten tricks you will not believe</a> <a href="/2">Cheap flights to everywhere</a></div>
<article><h1>Main &amp; text</h1>
<p>The first paragraph is long enough to count as content, with a <a href="/r">link</a> inside.
<p>An unclosed paragraph still ends at the next block, which is what error recovery buys here.</p></article>
<div>Copyright 2026 Example Corp.</div><footer><p>Imprint and privacy, in a long sentence that must not leak.</p></footer>"#;
    let t = main_text(page);
    assert!(t.starts_with("Main & text\n\nThe first paragraph") && t.contains("An unclosed paragraph"));
    for bad in ["Home", "Sponsored", "no<", "Copyright", "Imprint"] { assert!(!t.contains(bad), "leaked {bad}"); }
}

Thresholds (10 words, link density 0.33) are illustrative, not published values. Five entities only; no charset step.

Complexity and constants

• Time O(n); memory O(blocks), or O(DOM nodes) with a tree builder.
• Measured here (Apple M4 Pro, release build, one thread): 74-91 MB/s on a 27 MB synthetic page repeated 20,000 times, tokenizer plus scoring, no tree. Real pages (long attributes, deep nesting, entities) are slower.
• Cost drivers: DOM allocation, charset detection, fallback extractors tried, rendering.
• In trafilatura's benchmark its standard mode takes 2.9x the time of the simple baseline, fast 2.2x, precision 3.1x.

Efficiency

• Cost: a page of tens of KB takes order of 1 ms on one laptop core (from the MB/s above); the bound is allocation and memory bandwidth. Fetching costs more. A headless browser adds a full load: order of 100 ms to seconds and hundreds of MB of RAM per page (not measured).
• Levers:
  • Tokenizer pass without a DOM — no node allocation; 74-91 MB/s measured here — when only text and link density are needed.
  • Pick the mode — trafilatura fast F 0.920 at 2.2x versus standard F 0.926 at 2.9x (its benchmark) — when volume beats the last 0.6 points.
  • Static first, render on demand — order of 10-100x cheaper (not measured) — when most pages are server-rendered.
  • Skip unchanged pages — ETag / If-Modified-Since or a text hash — monitoring and re-crawls.
• Naive → standard → state of the art:
  • Strip all tags · precision 0.53, recall 0.99 (html_text, inscriptis in the trafilatura benchmark): all text, menus included
  • Block heuristics · jusText F 0.862, readability-lxml F 0.853
  • Rule cascade with fallbacks and metadata (trafilatura) · F 0.926 at 2.9x baseline

Implementation notes

• Pass the parser bytes plus the HTTP charset, not a wrongly decoded string.
• Keep the raw HTML (WARC): extractors improve, a fetch cannot be repeated.
• <td>, <li> and <br> must become separators, or "Home" and "News" fuse into "HomeNews".
• Record each block's tag path and byte range so chunks can cite a heading path.
• Normalize output (text normalization) before hashing or indexing.
• Fetch politely: robots rules.

Pitfalls

• Quadratic scans: lowercasing the rest of the document at each <script> made my first version over 30x slower (2 MB/s versus 91 MB/s measured); scan in place.
• A class hidden-xs is not the hidden attribute; CSS-hidden text in external stylesheets needs rendering to see.
• Comments, tables and code look like boilerplate (short, linky); decide per task.
• Docs sites put navigation in <div>, not <nav>; tag names alone leak sidebars.
• UTF-8 read as windows-1252 shows "é".
• Trafilatura's benchmark is mostly German articles; test your own domain.

Variants and alternatives

In production

• Search engines extract before the inverted index and BM25: menus add no terms.
• RAG over documentation sites: extract, chunk, embed.
• Firefox Reader View uses Readability.
• Price monitors read JSON-LD first, then visible text; compliance archives keep WARC plus extracted text.
• Pre-training data: Common Crawl's WET files hold extracted plain text; cleaner pipelines re-extract from WARC, then dedupe with MinHash.

Implementations

Sources

• Christian Kohlschütter, Peter Fankhauser, Wolfgang Nejdl, Boilerplate Detection using Shallow Text Features, WSDM 2010, pp. 441-450. paper · code
• Adrien Barbaresi, Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction, ACL 2021 (System Demonstrations). paper · code
• Trafilatura, Benchmarks and evaluation (results dated 2026-10-02). docs
• WHATWG, HTML Living Standard, Parsing HTML documents (determining the character encoding, tree construction). spec
• Mozilla, Readability.js. code
• Common Crawl, Get started (WARC, WAT, WET formats). docs