Unicode confusables

Deep dive

How it works

Parts:

• normaliser — NFC/NFKC (UAX #15) and case folding for identity.
• identifier profile — allowed characters (UTS #39 General Security Profile over UAX #31 XID).
• confusables table — confusables.txt: 0430 ; 0061 ; MA.
• script checker — resolved script set and restriction level.
• skeleton index — unique column holding the skeleton key.

1. Normalise for identity: UAX #31 caseless match is NFD then toNFKC_Casefold. Reject characters outside the profile (Default_Ignorable, Not_NFKC, not XID).
2. Restriction level (UTS #39 §5.2): ASCII-Only, Single Script, Highly Restrictive (Latin + Han/Hiragana/Katakana, Bopomofo or Hangul), Moderately Restrictive (Latin + one Recommended script, never Cyrillic or Greek), Minimally Restrictive, Unrestricted. Choose a level; reject looser strings.
3. Resolved script set: intersect each character's Script_Extensions (Common and Inherited count as all scripts; Han adds Jpan, Kore, Hanb). Empty means mixed-script.
4. internalSkeleton(X): NFD, delete Default_Ignorable_Code_Point, concatenate each character's prototype, NFD again. Mappings are idempotent, so no recursion. skeleton(X) = bidiSkeleton(LTR, X): first reorder by UAX #9 up to L2, apply L3 and L4 mirroring. Without R/AL characters this equals internalSkeleton.
5. X and Y are confusable iff skeletons are equal; the spec warns the test can be over-inclusive (transitivity).
6. Classify: single-script confusables share a script; mixed-script ones share none; whole-script ones are both single-script.
7. Index skeleton(name) uniquely; a collision rejects or flags the registration.
8. In domains, UTS #46 does the mapping first: Map, NFC, split at ., xn-- Punycode decode and validity criteria.

input ─► NFC/NFKC ─► profile ─► restriction level ─► skeleton ─► unique index
 raw        │ identity key (casefold)    │ mixed? reject          └ hit: look-alike

Code

Subset only: 16 entries of the real table and three scripts. Real systems generate the table from confusables.txt and use real script data. Caller supplies NFD input (std has no normaliser). Entries were checked against the 18.0.0 file.

use std::collections::HashMap;

/// SUBSET of confusables.txt (Unicode 18.0.0), sorted by source: source -> prototype.
pub const CONFUSABLES: &[(char, &str)] = &[
    ('\u{0030}', "O"), ('\u{0031}', "l"), ('\u{0049}', "l"), ('\u{0153}', "oe"),
    ('\u{03BF}', "o"), ('\u{0430}', "a"), ('\u{0435}', "e"), ('\u{043E}', "o"),
    ('\u{0440}', "p"), ('\u{0441}', "c"), ('\u{0443}', "y"), ('\u{0445}', "x"),
    ('\u{0455}', "s"), ('\u{0456}', "i"), ('\u{04CF}', "l"), ('\u{FF41}', "a"),
];

fn default_ignorable(c: char) -> bool { // subset of Default_Ignorable_Code_Point
    matches!(c as u32, 0xAD | 0x34F | 0x61C | 0x115F..=0x1160 | 0x17B4..=0x17B5
        | 0x180B..=0x180F | 0x200B..=0x200F | 0x202A..=0x202E | 0x2060..=0x206F
        | 0x3164 | 0xFE00..=0xFE0F | 0xFEFF | 0xFFA0 | 0xE0000..=0xE0FFF)
}

/// UTS #39 internalSkeleton. Input must already be NFD; the last NFD pass is
/// omitted because every prototype above is ASCII.
pub fn skeleton(nfd: &str) -> String {
    let mut out = String::with_capacity(nfd.len());
    for c in nfd.chars().filter(|&c| !default_ignorable(c)) {
        match CONFUSABLES.binary_search_by_key(&c, |&(k, _)| k) {
            Ok(i) => out.push_str(CONFUSABLES[i].1),
            Err(_) => out.push(c), // no entry: the character is its own prototype
        }
    }
    out
}

/// Caseless key: skeleton FIRST, lowercase after. The data maps '0'->'O' and
/// 'I'->'l'; lowercasing first would turn 'I' into 'i' and lose both.
pub fn key(nfd: &str) -> String { skeleton(nfd).to_lowercase() }

#[derive(Clone, Copy, PartialEq)]
enum Script { Latin, Greek, Cyrillic }

fn script(c: char) -> Option<Script> { // None = Common/Inherited; rough ranges
    match c as u32 {
        0x41..=0x5A | 0x61..=0x7A | 0xC0..=0xD6 | 0xD8..=0xF6 | 0xF8..=0x24F
        | 0xFF21..=0xFF3A | 0xFF41..=0xFF5A => Some(Script::Latin),
        0x370..=0x3FF => Some(Script::Greek),
        0x400..=0x52F => Some(Script::Cyrillic),
        _ => None,
    }
}

pub fn mixed_script(s: &str) -> bool { // empty resolved script set, simplified
    let mut seen = None;
    for sc in s.chars().filter_map(script) {
        match seen { None => seen = Some(sc), Some(p) if p != sc => return true, _ => {} }
    }
    false
}

/// Trojan Source: explicit embeddings/overrides (LRE..RLO) and isolates (LRI..PDI).
pub fn has_bidi_control(s: &str) -> bool {
    s.chars().any(|c| matches!(c as u32, 0x202A..=0x202E | 0x2066..=0x2069))
}

#[derive(Debug, PartialEq)]
pub enum Reject { Mixed, Bidi, Lookalike(String) }

#[derive(Default)]
pub struct Registry { by_key: HashMap<String, String> }

impl Registry {
    pub fn register(&mut self, name: &str) -> Result<(), Reject> {
        if has_bidi_control(name) { return Err(Reject::Bidi); }
        if mixed_script(name) { return Err(Reject::Mixed); }
        let k = key(name);
        if let Some(owner) = self.by_key.get(&k) { return Err(Reject::Lookalike(owner.clone())); }
        self.by_key.insert(k, name.to_string());
        Ok(())
    }
}

#[test]
fn spoofs() {
    let fake = "\u{440}\u{430}\u{443}\u{440}\u{430}l"; // "раураl": Cyrillic + Latin l
    assert_eq!(skeleton(fake), "paypal");
    assert!(mixed_script(fake) && !mixed_script("paypal"));
    let scope = "\u{455}\u{441}\u{43E}\u{440}\u{435}"; // all Cyrillic "ѕсоре"
    assert!(!mixed_script(scope)); // single-script check passes...
    assert_eq!(skeleton(scope), "scope"); // ...the skeleton does not
    assert_eq!(key("PayPaI"), key("paypal")); // capital I
    assert_ne!(skeleton("paypai"), skeleton("paypal")); // casefold-first would break this
    assert_eq!(key("g00gle"), key("google"));
    assert_eq!(skeleton("pay\u{200B}pal"), "paypal"); // zero-width space removed
    let mut r = Registry::default();
    assert_eq!(r.register("paypal"), Ok(()));
    assert_eq!(r.register(fake), Err(Reject::Mixed));
    assert_eq!(r.register("PayPaI"), Err(Reject::Lookalike("paypal".into())));
    assert_eq!(r.register("a\u{202E}b"), Err(Reject::Bidi));
}

Complexity and constants

• Skeleton: O(n) code points, one lookup each (sorted array, perfect hash or trie). Output can be longer than the input (up to 18 code points per character).
• Normalisation is linear but allocates; unbounded combining marks inflate it.
• Cost drivers: allocating a normalised copy, parsing the table at runtime, and pairwise comparison against all existing names.

Efficiency

Cost: one table lookup per code point plus one NFD and (identity) one NFKC_Casefold pass; memory-bound, nanoseconds per character in order of magnitude (not measured here). Registration cost is one indexed equality lookup; no syscall beyond the database write.

Levers:

• Unique index on the stored skeleton — O(1) per check instead of comparing against all N names (design, this page) — registries and signup.
• Compile confusables.txt into a static table at build time — no parsing or allocation at run time (load a precompiled table) — always.
• ASCII and Quick_Check fast paths: ASCII is unchanged by every normalisation form (UAX #15), and Quick_Check avoids normalising already-normal text — mostly-ASCII traffic. The skeleton still maps 0, 1, I.
• LTR fast path: with no R/AL characters bidiSkeleton equals internalSkeleton (UTS #39) — most Latin/Cyrillic names.
• Check at write time only; cap length and combining marks first (UAX #15 Stream-Safe: 30 non-starters; UTS #39 §5.4: more than 4 marks in a row) — public signup endpoints.

Naive → standard → state of the art:

• Compare each new name to every existing one with glyph or edit-distance heuristics · O(N·L) per signup.
• Skeleton column plus restriction level · O(L) per signup, one index probe.
• Add whole-script and TLD-aware checks, per-label display policy, bidi checks · same cost, fewer false negatives (browsers).

Implementation notes

• Run steps in the order: identity (NFKC_Casefold) for uniqueness; skeleton of the NFC/NFD original for look-alikes.
• ICU: uspoof_getSkeleton and uspoof_areConfusable with USPOOF_RESTRICTION_LEVEL (default Highly Restrictive); USPOOF_ANY_CASE is deprecated.
• Keep the Unicode version next to stored skeletons; rebuild the column when you update the data.
• Fonts matter: the data is conservative and font-independent, so some pairs are over- or under-flagged. Treat a hit as "review", not proof.

Pitfalls

• Lowercasing before the skeleton: PayPaI then skeletons to paypai, not paypal.
• Single-script check alone: all-Cyrillic "ѕсоре" passes.
• Comparing strings from different Unicode versions' skeletons.
• Allowing ZWJ, ZWNJ and variation selectors without context rules; the General Security Profile excludes them by default.
• Expecting sequence look-alikes such as rn vs m: the table maps one source character to a prototype, and UTS #39 §5.4 lists sequence-to-sequence cases (its Tamil example) as not handled.
• Skeleton of source identifiers does not stop Trojan Source: bidi controls sit in comments and strings, not identifiers.

Variants and alternatives

In production

• Browsers' IDN display (Chrome, Firefox) and ICU SpoofChecker users.
• rustc's identifier lints.
• Account, package and payee-name registries that store skeletons for uniqueness.

Implementations

Sources

• Unicode Consortium, UTS #39: Unicode Security Mechanisms (18.0.0). spec · confusables.txt
• Unicode Consortium, UAX #15: Unicode Normalization Forms. spec
• Unicode Consortium, UAX #31: Unicode Identifiers and Syntax. spec
• Unicode Consortium, UTS #46: Unicode IDNA Compatibility Processing. spec
• Unicode Consortium, UTS #55: Unicode Source Code Handling. spec
• Nicholas Boucher, Ross Anderson, Trojan Source: Invisible Vulnerabilities, USENIX Security 2023. paper · site
• Chromium docs, Internationalized Domain Names (IDN) in Google Chrome. doc
• Mozilla, IDN Display Algorithm. wiki
• rustc book, lint listings (confusable_idents, text_direction_codepoint_in_comment). doc