Intent classification and slot filling

Deep dive

How it works

Parts:

• Encoder — word features, or a pre-trained transformer shared by both heads.
• Intent head — softmax over intents on a sentence vector.
• Slot head — per-token softmax over BIO tags, optionally a CRF.
• Span decoder — repairs invalid BIO, joins tagged tokens into strings.
• Normalizer and resolver — dates, numbers (text-normalization), catalogue ids (entity-linking).
• Reject gate — max-probability threshold and/or an "out of scope" class.

Operations:

1. Tokenize; with subwords keep a word-to-first-subword map.
2. Encode once; the sentence vector feeds the intent head, each word's first-subword vector the slot head.
3. Train on the sum of both cross-entropies; mask padding and non-first subwords.
4. Decode: intent = argmax; tags = argmax per token (Viterbi with a CRF). Repair BIO: an I-x after anything but B-x/I-x becomes B-x.
5. Collect spans into slot -> text; keep slots the intent allows; normalize, resolve.
6. Gate: below threshold or out-of-scope: ask again or hand over.

utterance --> tokens --> encoder (shared) --+--> [CLS] vec --> intent softmax --> intent
                                            +--> word vecs --> BIO softmax/CRF --> tags
tags --> repair --> spans --> normalize --> resolve --> {intent, slots}
                  max-prob < tau  or  intent = oos  --> reject / ask again

A CRF on BERT performs "comparably" to softmax (Snips slot F1 96.7 vs 97.0, Chen et al.). Datasets: ATIS (flights, 21 intents), Snips (7 intents), CLINC150 (150 intents plus out-of-scope queries), MASSIVE (1 M utterances, 51 languages, 60 intents, 55 slots).

Code

No neural net: hashed bag-of-words softmax regression for the intent, a greedy perceptron BIO tagger, repair and span decoding, synthetic templates. Test cities oslo, lisbon, vienna never occur in training.

use std::collections::BTreeMap;
const D: usize = 1 << 14; // hash buckets
const INTENTS: [&str; 3] = ["book_flight", "weather", "reserve_table"];
const TAGS: [&str; 6] = ["O", "B-city", "I-city", "B-date", "I-date", "B-count"];
const TEMPLATES: [&[&str]; 3] = [
    &["book {count} tickets to {city} {date}", "i need a flight to {city} on {date}", "fly {count} people to {city} {date}", "cheap flights to {city} {date}"],
    &["what is the weather in {city} {date}", "will it rain in {city} {date}", "forecast for {city} {date}"],
    &["reserve a table for {count} {date}", "book a table for {count} people in {city} {date}", "table for {count} in {city} {date}"],
];
const CITIES: [&str; 8] = ["paris", "rome", "london", "new york", "los angeles", "berlin", "tokyo", "san francisco"];
const DATES: [&str; 6] = ["tomorrow", "today", "next friday", "this weekend", "monday", "next week"];
const COUNTS: [&str; 7] = ["2", "3", "4", "5", "8", "two", "six"];
struct Rng(u64);
impl Rng {
    fn below(&mut self, n: usize) -> usize { // xorshift64
        self.0 ^= self.0 << 13; self.0 ^= self.0 >> 7; self.0 ^= self.0 << 17;
        (self.0 % n as u64) as usize
    }
    fn pick<'a>(&mut self, v: &[&'a str]) -> &'a str { v[self.below(v.len())] }
}
fn hash(s: &str) -> usize {
    let h = s.bytes().fold(0xcbf29ce484222325u64, |h, b| (h ^ b as u64).wrapping_mul(0x100000001b3)); // FNV-1a
    (h >> 20) as usize % D
}
/// One synthetic utterance: (words, intent, BIO tag ids).
fn gen(rng: &mut Rng, cities: &[&str]) -> (Vec<String>, usize, Vec<usize>) {
    let intent = rng.below(3);
    let (mut words, mut tags) = (vec![], vec![]);
    for tok in rng.pick(TEMPLATES[intent]).split(' ') {
        let (fill, b) = match tok {
            "{city}" => (rng.pick(cities), 1), "{date}" => (rng.pick(&DATES), 3), "{count}" => (rng.pick(&COUNTS), 5), _ => (tok, 0),
        };
        for (j, w) in fill.split(' ').enumerate() { words.push(w.to_string()); tags.push(if b == 0 || j == 0 { b } else { b + 1 }); }
    }
    (words, intent, tags)
}
/// BIO repair: an I-x that does not continue a B-x/I-x of the same type becomes B-x.
fn repair(tags: &mut [usize]) {
    let ty = |t: usize| if t == 0 { "" } else { &TAGS[t][2..] };
    for i in 0..tags.len() {
        if TAGS[tags[i]].starts_with("I-") && (i == 0 || ty(tags[i - 1]) != ty(tags[i])) { tags[i] -= 1; }
    }
}
/// Decode (repaired) BIO tags into slot -> text; a repeated B- of the same slot overwrites.
fn spans(words: &[String], tags: &[usize]) -> BTreeMap<String, String> {
    let mut m = BTreeMap::new();
    for (w, &t) in words.iter().zip(tags).filter(|(_, &t)| t != 0) {
        let slot = TAGS[t][2..].to_string();
        if TAGS[t].starts_with("B-") { m.insert(slot, w.clone()); } else if let Some(v) = m.get_mut(&slot) { *v += &format!(" {w}"); }
    }
    m
}
/// Token features. The gazetteer knows only 5 of the 8 training cities, so the model must also learn context.
fn tok_feats(w: &[String], i: usize, prev: usize, intent: usize) -> [usize; 9] {
    let has = |l: &[&str], x: &str| l.iter().any(|c| c.split(' ').any(|p| p == x));
    let (cur, pw) = (w[i].as_str(), if i > 0 { w[i - 1].as_str() } else { "<s>" });
    let (nw, digit) = (w.get(i + 1).map_or("</s>", |s| s.as_str()), cur.bytes().all(|b| b.is_ascii_digit()));
    let g = if has(&CITIES[..5], cur) { "city" } else if has(&DATES, cur) { "date" } else { "-" };
    [hash(&format!("w={cur}")), hash(&format!("p={pw}")), hash(&format!("n={nw}")), hash(&format!("d={digit}")), hash(&format!("g={g}")),
     hash(&format!("t={prev}")), hash(&format!("t+p={prev}{pw}")), hash(&format!("p+g={pw}{g}")), hash(&format!("i={intent}{pw}"))] // intent feeds the tagger
}
fn ifeats(w: &[String]) -> Vec<usize> {
    (0..w.len()).flat_map(|i| [hash(&w[i]), hash(&format!("{} {}", if i > 0 { &w[i - 1] } else { "<s>" }, w[i]))]).collect()
}
struct Nlu { lr: Vec<f32>, perc: Vec<f32> } // softmax-regression weights [f*3+k]; perceptron weights [tag*D+f]
impl Nlu {
    fn intent_scores(&self, w: &[String]) -> [f32; 3] {
        let mut z = [0f32; 3];
        for f in ifeats(w) { for k in 0..3 { z[k] += self.lr[f * 3 + k]; } }
        let m = z.iter().cloned().fold(f32::MIN, f32::max); // subtract max for stability
        let s: f32 = z.iter().map(|v| (v - m).exp()).sum();
        z.map(|v| (v - m).exp() / s)
    }
    fn sgd(&mut self, w: &[String], y: usize, lr: f32) {
        let p = self.intent_scores(w); // d loss / d logit_k = p_k - [k == y]
        for f in ifeats(w) { for k in 0..3 { self.lr[f * 3 + k] -= lr * (p[k] - (k == y) as u8 as f32); } }
    }
    fn best_tag(&self, f: &[usize; 9]) -> usize {
        let s = |t: usize| f.iter().map(|&h| self.perc[t * D + h]).sum::<f32>();
        (0..TAGS.len()).max_by(|&a, &b| s(a).partial_cmp(&s(b)).unwrap()).unwrap()
    }
    fn parse(&self, text: &str) -> (&'static str, BTreeMap<String, String>) {
        let w: Vec<String> = text.split_whitespace().map(String::from).collect();
        let p = self.intent_scores(&w); let intent = (0..3).max_by(|&a, &b| p[a].partial_cmp(&p[b]).unwrap()).unwrap();
        let mut tags = vec![0usize; w.len()];
        for i in 0..w.len() { tags[i] = self.best_tag(&tok_feats(&w, i, if i > 0 { tags[i - 1] } else { 0 }, intent)); } // greedy
        repair(&mut tags);
        (INTENTS[intent], spans(&w, &tags))
    }
}
fn main() {
    let mut rng = Rng(0x9E3779B97F4A7C15); let train: Vec<_> = (0..3000).map(|_| gen(&mut rng, &CITIES)).collect();
    let all = [CITIES.as_slice(), &["oslo", "lisbon", "vienna"]].concat(); // 3 of 11 test cities never seen in training
    let test: Vec<_> = (0..1000).map(|_| gen(&mut rng, &all)).collect(); let mut m = Nlu { lr: vec![0.0; D * 3], perc: vec![0.0; D * TAGS.len()] };
    for epoch in 0..5 {
        for (w, y, tags) in &train {
            m.sgd(w, *y, 0.3 / (1.0 + epoch as f32));
            for i in 0..w.len() { // perceptron, gold previous tag and gold intent (teacher forcing)
                let f = tok_feats(w, i, if i > 0 { tags[i - 1] } else { 0 }, *y);
                let p = m.best_tag(&f);
                if p != tags[i] { for &h in &f { m.perc[tags[i] * D + h] += 1.0; m.perc[p * D + h] -= 1.0; } }
            }
        }
    }
    let ok = |f: &dyn Fn(&(Vec<String>, usize, Vec<usize>)) -> bool| test.iter().filter(|u| f(u)).count() as f64 / test.len() as f64;
    let (acc, em) = (ok(&|(w, y, _)| INTENTS[*y] == m.parse(&w.join(" ")).0), ok(&|(w, _, t)| m.parse(&w.join(" ")).1 == spans(w, t)));
    println!("held-out intent accuracy {acc:.3}, exact slot match {em:.3}");
    assert!(acc > 0.95 && em > 0.9);
    for (text, intent, slots) in [("cheap flights to oslo tomorrow", "book_flight", "city=oslo date=tomorrow"),
        ("table for 4 in new york this weekend", "reserve_table", "city=new york count=4 date=this weekend")] {
        let (got, map) = m.parse(text);
        let shown: Vec<String> = map.iter().map(|(k, v)| format!("{k}={v}")).collect();
        assert_eq!((got, shown.join(" ")), (intent, slots.to_string()));
    }
    let mut t = vec![4, 3, 2, 0, 4]; // I-date first, I-city after B-date, I-date after O
    repair(&mut t); assert_eq!(t, vec![3, 3, 1, 0, 3]); // all become B-
    let texts: Vec<String> = test.iter().map(|(w, _, _)| w.join(" ")).collect(); let t0 = std::time::Instant::now();
    let n: usize = (0..20).map(|_| texts.iter().map(|s| m.parse(s).1.len()).sum::<usize>()).sum();
    println!("{:.0} utterances/s, one thread (checksum {n})", 20.0 * texts.len() as f64 / t0.elapsed().as_secs_f64());
}

Output on an Apple M4 Pro, one thread: held-out intent accuracy 1.000, exact slot match 1.000 (easy synthetic templates); about 114 k to 163 k utterances/s over twelve runs, typically 155 k.

Complexity and constants

• Linear model: O(tokens x features x (intents + tags)); weights 192 KiB + 384 KiB, resident in L2.
• Encoder: about 2 x parameters x tokens FLOPs; utterances are short, so weight reads dominate, not attention.
• Cost drivers: hashing (linear), weight bytes (encoder), round trip and schema tokens (LLM).

Efficiency

• Cost: toy model about 6.5 µs per utterance (measured, about 155 k/s on one M4 Pro core, bound by format! allocation and hashing). BERT-Base, 12 tokens: order of 2 GFLOPs (my arithmetic: 2 x ~85 M non-embedding parameters x 12). LLM: a network call priced per token (not measured).
• Levers:
  • one shared encoder pass for both heads — one forward pass instead of two (arithmetic) — joint models.
  • cascade: linear model first, encoder or LLM below the threshold — cost scales with the escalated share — few head intents dominate traffic.
  • cache by normalized utterance — a lookup instead of a forward pass — repeated commands.
• Naive → standard → state of the art:
  • LLM prompt per utterance with the action schema · per-token price, network latency.
  • joint fine-tuned encoder, one pass · Snips frame accuracy 92.8 % (Chen et al.).
  • cache, linear model, joint encoder, constrained-LLM fallback · cheap-rung cost for the confident majority.

Implementation notes

• Label only the first subword of each word, mask the rest, read the first-subword vector at inference (Chen et al.).
• Tag surface text ("next friday"), normalize afterwards against a request-supplied reference date.
• Out-of-scope: an oos class from real off-topic queries plus a max-probability threshold, calibrated.
• Report intent accuracy, slot F1 and frame accuracy (evaluation-metrics); keep an out-of-scope test set.

Pitfalls

• Gazetteer over-reliance. Measured on this toy, held-out exact slot match: 0.756 with all 8 training cities in the gazetteer (unseen cities missed); 0.971 listing only 5; 1.000 after also giving the tagger the predicted intent ("for oslo" is a count in one intent, a city in another).
• Per-token softmax can emit an I- without B-: repair or constrain.
• Multi-intent requests ("book it and text my wife") fit neither head.
• ASR errors hit names and numbers hardest.

Variants and alternatives

In production

• Smart speakers, phones and car head units (MASSIVE holds assistant utterances).
• Banking and support bots, IT helpdesks (ticket category, host), travel search and search-query understanding use the same frame.

Implementations

Sources

• Liu, Lane, Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling, Interspeech 2016. paper
• Chen, Zhuo, Wang, BERT for Joint Intent Classification and Slot Filling, 2019 (preprint). paper
• Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, EMNLP-IJCNLP 2019 (CLINC150). paper · data
• FitzGerald et al., MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages, 2022. paper · data
• Bunk, Varshneya, Vlasov, Nichol, DIET: Lightweight Language Understanding for Dialogue Systems, 2020. paper · code