Language identification

Deep dive

How it works

In: UTF-8 text of any length · Out: ISO 639 code and confidence, or und · State: per-language log-probability tables (read-only) · Cost: O(n · L), about 1.3 µs per 63-byte text with 4 languages (measured, see Efficiency)

1. Decode, normalise. Valid text first (charset: see below). Lowercase; strip URLs, @handles, #tags, digits, punctuation (Text normalization); they carry no language.
2. Script stage. Count letters per Unicode script. Any kana means Japanese; mostly Greek, Hangul or Thai maps to one language; Han alone is Chinese or Japanese and needs a model. CLD2 uses the same shortcut for scripts that map one-to-one to a language.
3. Too little text. Under a few letters, return und.
4. Features. Padded character n-grams (1-3 here; Cavnar and Trenkle 1-5; langid.py byte n-grams 1-4), hashed to a fixed table (Feature hashing).
5. Score. Rank profiles: sum each n-gram's rank displacement against a language profile ("out-of-place" distance; smallest wins). Probabilistic: sum log P(gram | language) (Naive Bayes). fastText-style: average n-gram embeddings, one softmax layer; CLD3 adds a hidden layer.
6. Decide. Softmax gives a posterior; below a threshold (0.9 here) or with a small margin, return und.
7. Per span (optional). Mixed documents: split on script changes and sentence ends, run 2-6 per span, merge neighbours with equal labels.

Code

Std-only Rust: script stage, normaliser, hashed 1-3-gram naive Bayes trained on two short sentences per language that I wrote for this page (en, de, fr, es). Compiled with cargo run --release; the asserts pass.

use std::time::Instant;

const B: usize = 1 << 14; // hash buckets per language

/// Stage 1, Unicode ranges: 0 Latin, 1 Greek, 2 Kana, 3 Hangul, 4 other (Han, Cyrillic, Arabic, ...).
fn script(c: char) -> usize {
    match c as u32 {
        0x41..=0x5A | 0x61..=0x7A | 0xC0..=0x24F | 0x1E00..=0x1EFF => 0,
        0x370..=0x3FF | 0x1F00..=0x1FFF => 1,
        0x3040..=0x30FF => 2,
        0xAC00..=0xD7AF | 0x1100..=0x11FF => 3,
        _ => 4,
    }
}

/// Lowercase letters only; URLs, @handles, #tags, digits and punctuation dropped.
fn normalise(s: &str) -> String {
    let mut out = String::from(" ");
    for w in s.split_whitespace().filter(|w| !w.starts_with(['@', '#']) && !w.starts_with("http")) {
        let t: String = w.chars().filter(|c| c.is_alphabetic()).flat_map(char::to_lowercase).collect();
        if !t.is_empty() { out += &t; out.push(' '); }
    }
    out
}

/// Hashed character 1-, 2-, 3-grams (FNV-1a over code points).
fn grams(text: &str, mut f: impl FnMut(usize)) {
    let cs: Vec<char> = text.chars().collect();
    for n in 1..=3 {
        for w in cs.windows(n).filter(|w| n > 1 || w[0] != ' ') {
            let h = w.iter().fold(2166136261u32, |h, &c| (h ^ c as u32).wrapping_mul(16777619));
            f(h as usize % B);
        }
    }
}

struct Model { langs: Vec<&'static str>, logp: Vec<Vec<f32>> }

impl Model {
    fn train(data: &[(&'static str, &str)]) -> Model {
        let (mut langs, mut logp) = (vec![], vec![]);
        for &(l, text) in data {
            let mut c = vec![0.5f32; B]; // add-0.5 smoothing
            grams(&normalise(text), |h| c[h] += 1.0);
            let tot: f32 = c.iter().sum();
            langs.push(l);
            logp.push(c.iter().map(|x| (x / tot).ln()).collect());
        }
        Model { langs, logp }
    }

    /// (language, posterior); "und" if too little text, an unmodelled script or posterior < 0.9.
    fn detect(&self, text: &str) -> (&'static str, f32) {
        let mut n = [0usize; 5];
        for c in text.chars().filter(|c| c.is_alphabetic()) { n[script(c)] += 1; }
        if n.iter().sum::<usize>() < 3 { return ("und", 0.0); }
        if n[2] > 0 { return ("ja", 1.0); } // any kana: Japanese, even if mostly Han
        match (0..5).max_by_key(|&i| n[i]).unwrap() {
            1 => return ("el", 1.0), 3 => return ("ko", 1.0), 4 => return ("und", 0.0), _ => {}
        }
        let mut s = vec![0f32; self.langs.len()];
        grams(&normalise(text), |h| for (i, l) in self.logp.iter().enumerate() { s[i] += l[h]; });
        let m = s.iter().cloned().fold(f32::MIN, f32::max);
        let p = 1.0 / s.iter().map(|x| (x - m).exp()).sum::<f32>(); // softmax of the winner
        let best = s.iter().position(|&x| x == m).unwrap();
        if p < 0.9 { ("und", p) } else { (self.langs[best], p) }
    }
}

const EN: &str = "She has been working at the library since last year and she likes the quiet rooms. \
    There are many people in the market who sell fresh bread, fruit and cheese to the visitors.";
const DE: &str = "Sie arbeitet seit letztem Jahr in der Bibliothek und mag die ruhigen Räume sehr. \
    Auf dem Markt gibt es viele Leute, die frisches Brot, Obst und Käse an die Besucher verkaufen.";
const FR: &str = "Elle travaille à la bibliothèque depuis l'année dernière et elle aime beaucoup les salles tranquilles. \
    Au marché, beaucoup de gens vendent du pain frais, des fruits et du fromage aux visiteurs.";
const ES: &str = "Ella trabaja en la biblioteca desde el año pasado y le gustan mucho las salas tranquilas. \
    En el mercado hay mucha gente que vende pan fresco, fruta y queso a los visitantes.";

fn main() {
    let m = Model::train(&[("en", EN), ("de", DE), ("fr", FR), ("es", ES)]);
    let held_out = [
        ("We could not find the train, so the whole family waited at the market.", "en"),
        ("Der Bruder wartet am Bahnhof, weil der Zug heute später kommt.", "de"),
        ("Nous attendons le dîner avec toute la famille, mais personne ne vient.", "fr"),
        ("La familia espera en el mercado porque el tren llega más tarde.", "es"),
        ("@anna http://x.co/12 Dein Plan ist gut, aber das Geld reicht nicht!!! 123", "de"),
        ("Καλημέρα σε όλους", "el"), ("今日はいい天気ですね", "ja"),
        ("Привет, как дела?", "und"), ("ok", "und"), // no Cyrillic model here; too short
    ];
    for (t, want) in held_out { assert_eq!(m.detect(t).0, want, "{t}"); }
    let (l, p) = m.detect("chat"); // French "cat": one word, confidently wrong
    assert!(l == "en" && p > 0.9);
    let t0 = Instant::now();
    for _ in 0..200_000 { std::hint::black_box(m.detect(held_out[1].0)); }
    println!("{:.2} us per 63-byte text", t0.elapsed().as_secs_f64() / 200_000.0 * 1e6);
}

Held-out sentences pass, including one with a handle, URL and digits. The failure case is the point: chat (French for "cat") is labelled English with posterior 0.99, and hotel taxi Spanish with 0.999 (measured, this model). Naive Bayes treats n-grams as independent, so a few of them give a near-1 posterior. For short input, length is the reliable confidence signal.

Charset handoff

This stage takes decoded text (fastText's models expect UTF-8). Byte order mark, HTTP charset, the <meta> prescan and a detector as last resort are the decoding stage of HTML main-text extraction: decode there, pass valid UTF-8 here. Mojibake from a wrong charset yields plausible letters of the wrong language, or und.

Complexity and constants

• Time: O(n · L) for naive Bayes; with L = 176 prune candidates by script, or use a linear model whose cost is set by its hidden size.
• Memory: 4 bytes × buckets per language (64 KB at 16,384 buckets here); lid.176 is 126 MB, compressed 917 kB.
• About 3 n-grams per character (orders 1-3), each a random read into a cache-resident table per language.
• Cost drivers: input length, candidate languages, table size vs cache.

Efficiency

• Cost: measured on an Apple M4 Pro, one thread, release build: 1.2-1.5 µs per 63-byte text (≈ 45 MB/s), 4 languages, 3 n-gram orders, 16,384 buckets, 200,000 repeats of one text. Bound: table reads × languages; linear in L, so 40 languages cost order of 10x more unless pruned.
• Levers:
  • script pre-check — skips the model for one-to-one scripts (CLD2 README) — Greek, Hangul, Thai, kana
  • restrict candidate languages — cost falls with L (langid.py set_languages) — deployment knows its set
  • quantize and prune — .bin 126 MB to .ftz 917 kB, .bin slightly more accurate (fastText page) — memory-bound or embedded use
  • truncate long input — cost is linear in n, the label is stable after a few hundred characters (order of; measure on your data) — documents
  • reuse the already normalised text of the pipeline — one pass instead of two — crawl pipelines
• Naive → standard → state of the art:
  • rank profiles · build and sort a profile per text, compare with every language (Cavnar and Trenkle)
  • multinomial naive Bayes, byte n-grams 1-4, feature selection · 97 languages (langid.py)
  • script routing plus quantized n-gram tables, or hashed-embedding models · CLD2 (over 80 languages), CLD3, fastText lid.176 (176)

Implementation notes

• Train on the domain you serve with the same normalisation at train and test time; langid.py selects features by information gain across domains for this reason.
• Close pairs: add word-level evidence (CLD2 scores distinctive words and word pairs for Malay/Indonesian and Spanish/Portuguese/Galician). Bosnian, Croatian and Serbian share most vocabulary and Serbian uses Cyrillic and Latin, so script does not separate them: merge into one label if downstream steps allow, else use locale or domain as a prior.
• Mixed documents: CLD2 returns the top three languages with shares of text bytes and optional spans; lingua advertises mixed-language detection. Detect per span, never average over a bilingual document.
• Return und on low confidence, mul only when several languages are known to occur; keep the script as fallback label (und-Cyrl).
• Calibrate: map margin and length to an empirical error rate on labelled data (Calibration).

Pitfalls

• Trusting a naive Bayes posterior on short input (shown above).
• Boilerplate (menus, cookie banners, code) votes for English: extract the main text first.
• Names, brands and loanwords are no evidence about the surrounding language.
• Homoglyph text (Latin letters swapped for Cyrillic) breaks script counts; see Unicode confusables.
• Same-source training and testing overstates accuracy; the Jauhiainen survey discusses evaluation.
• Unsupported languages land on the nearest supported one: keep an "other" class or a distance threshold.

Variants and alternatives

In production

CLD2 (Chromium's embedded detector, per its PyPI description), CLD3, fastText lid.176 and langid.py are the common off-the-shelf choices; whatlang and lingua are the Rust ones (below).

Implementations

Sources

• W. B. Cavnar, J. M. Trenkle, N-Gram-Based Text Categorization, Symposium on Document Analysis and Information Retrieval (SDAIR), 1994. paper
• M. Lui, T. Baldwin, langid.py: An Off-the-shelf Language Identification Tool, ACL 2012 System Demonstrations. paper · code
• A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of Tricks for Efficient Text Classification, 2016. paper · lid.176 models
• T. Jauhiainen, M. Lui, M. Zampieri, T. Baldwin, K. Lindén, Automatic Language Identification in Texts: A Survey, JAIR 65, 2019. paper · arXiv
• CLD2 and CLD3 READMEs; whatlang and lingua READMEs.