Text normalization

Deep dive

How it works

In: raw text plus locale · Out: normalized text and a span map back to source offsets · State: lexicons, class grammars, speller dictionary (read-only) · Cost: O(n) in input bytes; 61-316 MB/s measured here (see Efficiency)

1. Repair encoding: fix mojibake only when clearly better (✔ → ✔, ftfy); decode stray HTML entities.
2. Unicode form: NFC for storage; NFKC and folds only for match keys (Unicode text tokenization, UAX #15). Collapse whitespace, drop control characters.
3. Tag spans by class; numbers before spelling.
4. Canonical value or words: 1.234,56 € → 1234.56 EUR, 0.5 l → 500 ml, $12.50 → twelve dollars fifty cents. A missing locale is an error, not a default.
5. Expand abbreviations and slang; ambiguous ones (St, MS) need context or stay.
6. Spell-correct unprotected PLAIN tokens: delete up to d characters from dictionary words and the token, intersect, rank by edit distance then frequency (SymSpell, Edit distance).
7. Skeleton (UTS #39): NFD, drop default-ignorable code points, map each to its confusables.txt prototype, NFD again; equal skeletons are confusable (Unicode confusables).
8. Emit text and span map; model outputs (entities, flags) map back via to_src.

source  "Dr.  Smith paid $12.50 for 2kg"
tag      PLAIN  PLAIN  MONEY       MEASURE
rewrite  doctor Smith paid twelve dollars fifty cents for two kilograms
map      "fifty" --to_src--> "$12.50"      "Smith" --> same bytes, 1:1

ITN is ambiguous: twelve fifty is $12.50, 12:50 or 1250; context or weights choose. Production TTS compiles tagger and verbalizer into weighted transducers (Finite-state transducer): Google's Kestrel (open source: Sparrowhawk) and NVIDIA NeMo, on Pynini and OpenFst. Sparrowhawk links input tokens, positions and output words, like the span map. RNNs read 2 mA as "two units" and £900 million as euros (Sproat & Jaitly), so an FST filter limits outputs to the grammar's verbalizations, or a model only scores the grammar's candidates (NeMo hybrid).

Code

TTS rule pipeline with span map: whitespace, abbreviations, cardinals to 999 999, money, decimals. Std only.

use std::ops::Range;
/// One piece of output and the source bytes it came from. `exact` = copied byte for byte,
/// so a position inside it maps to the same position in the source.
pub struct Span { pub out: Range<usize>, pub src: Range<usize>, pub exact: bool }
pub struct Normalized { pub text: String, pub spans: Vec<Span>, pending: Option<Range<usize>> }

const ONES: [&str; 20] = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten",
    "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen"];
const TENS: [&str; 10] = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"];
const ABBR: [(&str, &str); 4] = [("dr", "doctor"), ("mr", "mister"), ("approx", "approximately"), ("kg", "kilograms")];
fn below_1000(n: u32) -> String {
    match n {
        0..=19 => ONES[n as usize].to_string(),
        20..=99 => if n % 10 == 0 { TENS[(n / 10) as usize].into() } else { format!("{}-{}", TENS[(n / 10) as usize], ONES[(n % 10) as usize]) },
        _ => if n % 100 == 0 { format!("{} hundred", ONES[(n / 100) as usize]) } else { format!("{} hundred {}", ONES[(n / 100) as usize], below_1000(n % 100)) },
    }
}
/// 0..=999_999 as words; longer or zero-padded digit strings (phone numbers, ids) are read digit by digit.
pub fn speak_number(d: &str) -> String {
    if d.len() > 6 || (d.len() > 1 && d.starts_with('0')) {
        return d.bytes().map(|c| ONES[(c - b'0') as usize]).collect::<Vec<_>>().join(" ");
    }
    let n: u32 = d.parse().unwrap();
    if n < 1000 { return below_1000(n); }
    let rest = n % 1000;
    format!("{} thousand{}", below_1000(n / 1000), if rest > 0 { format!(" {}", below_1000(rest)) } else { String::new() })
}
impl Normalized {
    fn push(&mut self, s: &str, src: Range<usize>, exact: bool) {
        if let Some(r) = self.pending.take() { // a collapsed whitespace run is written only before more content
            if !self.text.is_empty() { self.add(" ", r, false); }
        }
        self.add(s, src, exact);
    }
    fn add(&mut self, s: &str, src: Range<usize>, exact: bool) {
        let o = self.text.len();
        self.text.push_str(s);
        if exact { // merge adjacent copied runs into one span
            if let Some(l) = self.spans.last_mut() {
                if l.exact && l.src.end == src.start { l.out.end = self.text.len(); l.src.end = src.end; return; }
            }
        }
        self.spans.push(Span { out: o..self.text.len(), src, exact });
    }
    /// Source range (char-aligned) that produced output byte `pos`.
    pub fn to_src(&self, src: &str, pos: usize) -> Option<Range<usize>> {
        let k = self.spans.partition_point(|s| s.out.end <= pos);
        let s = self.spans.get(k).filter(|s| s.out.start <= pos)?;
        Some(if s.exact {
            let mut p = s.src.start + pos - s.out.start;
            while !src.is_char_boundary(p) { p -= 1; }
            p..p + src[p..].chars().next()?.len_utf8()
        } else { s.src.clone() })
    }
}
/// Digits with `,ddd` groups starting at `i`; returns (end, digits without commas).
fn digits(b: &[u8], mut i: usize) -> (usize, String) {
    let mut d = String::new();
    loop {
        while i < b.len() && b[i].is_ascii_digit() { d.push(b[i] as char); i += 1; }
        if i + 3 < b.len() && b[i] == b',' && b[i + 1..i + 4].iter().all(u8::is_ascii_digit) { i += 1 } else { return (i, d) }
    }
}
pub fn normalize(src: &str) -> Normalized {
    let (b, n) = (src.as_bytes(), src.len());
    let mut o = Normalized { text: String::with_capacity(n + n / 4), spans: Vec::new(), pending: None };
    let mut i = 0;
    while i < n {
        let c = src[i..].chars().next().unwrap();
        let w = c.len_utf8();
        if c.is_whitespace() || c.is_control() { // collapse a run into one space
            let j = i + src[i..].find(|x: char| !(x.is_whitespace() || x.is_control())).unwrap_or(n - i);
            o.pending = Some(o.pending.take().map_or(i..j, |p| p.start..j));
            i = j;
        } else if c.is_ascii_digit() || (c == '$' && i + 1 < n && b[i + 1].is_ascii_digit()) {
            let money = c == '$';
            let (mut j, d) = digits(b, i + money as usize);
            let mut words = speak_number(&d);
            if money { // "$12.50": dollars, then exactly two digits of cents
                let cents = (j + 2 < n && b[j] == b'.' && b[j + 1].is_ascii_digit() && b[j + 2].is_ascii_digit()).then(|| src[j + 1..j + 3].parse::<u32>().unwrap());
                let unit = |n: &str, one: &str| format!("{} {}{}", speak_number(n), one, if n == "1" { "" } else { "s" });
                words = unit(&d, "dollar");
                if let Some(c) = cents { j += 3; words = match (d.as_str(), c) { (_, 0) => words, ("0", _) => unit(&c.to_string(), "cent"), _ => format!("{} {}", words, unit(&c.to_string(), "cent")) }; }
            } else if j + 1 < n && b[j] == b'.' && b[j + 1].is_ascii_digit() { // "3.14": digits after the point read one by one
                let mut k = j + 1;
                while k < n && b[k].is_ascii_digit() { k += 1; }
                words = format!("{} point {}", words, speak_number(&format!("0{}", &src[j + 1..k])).strip_prefix("zero ").unwrap());
                j = k;
            }
            if j < n && b[j].is_ascii_alphabetic() { words.push(' '); } // "2kg" -> "two kilograms"
            o.push(&words, i..j, false);
            i = j;
        } else if c.is_alphabetic() {
            let j = i + src[i..].find(|x: char| !x.is_alphabetic()).unwrap_or(n - i);
            let hit = ABBR.iter().find(|(a, _)| src[i..j].eq_ignore_ascii_case(a));
            match hit {
                Some((_, full)) => { let e = j + (j < n && b[j] == b'.') as usize; o.push(full, i..e, false); i = e; }
                None => { o.push(&src[i..j], i..j, true); i = j; }
            }
        } else { o.push(&src[i..i + w], i..i + w, true); i += w; }
    }
    o
}
#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn money_and_offsets() {
        let s = "  Dr.  Smith paid $12.50 \t for 2kg;  π≈3.14 ";
        let n = normalize(s);
        assert_eq!(n.text, "doctor Smith paid twelve dollars fifty cents for two kilograms; π≈three point one four");
        let at = |w: &str| &s[n.to_src(s, n.text.find(w).unwrap()).unwrap()];
        assert_eq!((at("fifty"), at("Smith"), at("π")), ("$12.50", "S", "π"));
        assert_eq!(speak_number("21000"), "twenty-one thousand");
        assert_eq!(normalize("$1").text, "one dollar");
    }
}

Complexity and constants

• One scan, O(n); cost is allocation and one span per changed or separated piece.
• Span: 40 bytes (size_of, measured). Number-heavy text: 8.6 million spans for 23.2 MB, about 15 map bytes per input byte (computed).
• SymSpell README: 25 deletes cover a 5-letter word at distance 3, against about 3 million candidates with inserts/replaces; price: a RAM index of all dictionary deletes.

Efficiency

• Cost: measured by me, one Apple M4 Pro core, release build, input in memory: 60-63 MB/s on 23.2 MB of number-heavy text (a number, amount or abbreviation every ~15 bytes, output 2.2× longer); 306-316 MB/s on 23 MB of prose (3 runs, best of 5). Bound: allocation and branching per token. Neural normalizers add model inference per sentence (no figure).
• Levers:
  • Classify only tokens with digits, symbols or table hits — prose ran 5× faster than number-heavy text (measured) — when most text is PLAIN.
  • Gap map: only changed pieces, two u32 each — less map memory (design estimate) — long documents with offsets.
  • One compiled transducer for all grammars — one pass whatever the rule count (Sparrowhawk/NeMo design) — hundreds of rules.
  • Precomputed delete index — README claims a million times faster than naive candidates — dictionary fits in RAM.
• Naive → standard → state of the art:
  • chain of regex replace passes, offsets lost · O(rules × n), order bugs
  • tagger + per-class grammar compiled to WFSTs, span links kept · O(n), auditable
  • neural tagger or scorer proposes, WFST grammar constrains · inference plus grammar, no out-of-grammar outputs

Implementation notes

• Order: repair → Unicode → whitespace → tag → numbers/units → abbreviations → spelling; else 12.50 gets "corrected".
• Keep display text, match key and skeleton apart; store the pipeline and Unicode versions with keys.
• Lexicons per locale and domain; test idempotence.

Pitfalls

• Wrong-meaning verbalization (2 mA as "two units", 16 ГБ as "sixteen hours", Sproat & Jaitly) is worse than leaving the token.
• Locale: 1.000 is one thousand in German; 03/04 is ambiguous. Context-free expansion: St street or saint, MS sclerosis or mitral stenosis.
• Spelling correction on names, codes, rare terms (dysphasia ↔ dysphagia) flips meaning; protect classes, use a domain dictionary.
• NFKC on stored text changes it (² → 2); skeletons flag look-alikes, and legitimate mixed-script words exist.
• The toy has no € or 12,50 and eats the dot of a sentence-final kg..

Variants and alternatives

In production

Google TTS (Kestrel), NeMo text processing for TTS and ITN of ASR output, ftfy in text cleaning, SymSpell ports in query correction, confusable skeletons in abuse filters.

Implementations

Sources

• R. Sproat, N. Jaitly, RNN Approaches to Text Normalization: A Challenge, 2016. paper · data
• H. Zhang et al., Neural Models of Text Normalization for Speech Applications, Computational Linguistics 45(2), 2019. paper
• P. Ebden, R. Sproat, The Kestrel TTS text normalization system, Natural Language Engineering 21(3), 2015, pp. 333-353.
• Unicode UAX #15 (normalization forms) and UTS #39 (skeleton, confusables).
• W. Garbe, SymSpell and blog; R. Speer, ftfy.
• NeMo-text-processing · Sparrowhawk · OpenFst