Model routing and cascades

Deep dive

How it works

Parts:

• Router — maps a request to a model choice: rules, a learned classifier, or the cascade controller.
• Model pool — ordered tiers (small, medium, large), each with price, latency and limits.
• Quality signal — token log-probs, a verifier or judge score, agreement of repeated samples, or a task check (JSON parses, tests pass).
• Threshold — turns the signal into "accept" or "escalate"; fitted on held-out labelled traffic.
• Fallback layer — retries, cooldowns and failover to another deployment on errors, rate limits or outages.
• Telemetry — model, cost, escalation flag and outcome per request, for tuning and drift detection.

Operations:

1. Static rules first: by task type (classification vs open reasoning), input length or context need, whether tools are required, user tier. Cheapest router; no extra model call.
2. Learned router (optional): train a classifier to predict "the small model is good enough", from preference data (RouteLLM routers: mf, bert, causal_llm, sw_ranking) or predicted query difficulty (Hybrid LLM). Send to the large model if the score passes a threshold. Cost: one cheap forward pass before any LLM call.
3. Cascade (alternative or addition): call the small model; compute a confidence; if it is below t, call the next tier and return that answer (FrugalGPT learns which model combinations to use).
4. Choose t on a held-out set: sweep t, record accuracy and cost, take the cheapest point meeting a target (quality loss budget vs always-large).
5. Fall back on failure: classify the error (rate limit, timeout, overloaded, context too long); retry with backoff, cool the deployment down after repeated failures (circuit breaker), then fail over to another provider or tier.
6. Log and re-tune: escalation rate, cost per solved task, regret against always-large on a sampled audit; refit thresholds on model or prompt change.

request -> [rules] -> [router score] --easy--> small --conf>=t--> answer
                          |hard                  |conf<t
                          v                       v
                        large <-------------------+      error/429 -> breaker -> other deployment

Code

Cascade on synthetic data (no real model is called): each query has a hidden difficulty; the small model's correctness and confidence follow from it; escalated queries pay for both calls. Costs are assumed units (large = 20x small).

const C_SMALL: f64 = 1.0;
const C_LARGE: f64 = 20.0; // assumed price ratio

struct Rng(u64);
impl Rng {
    fn next(&mut self) -> f64 { // xorshift64*, uniform in [0,1)
        self.0 ^= self.0 >> 12; self.0 ^= self.0 << 25; self.0 ^= self.0 >> 27;
        (self.0.wrapping_mul(0x2545F4914F6CDD1D) >> 11) as f64 / (1u64 << 53) as f64
    }
}

struct Query { small_ok: bool, large_ok: bool, conf: f64 }

fn make_queries(n: usize, seed: u64) -> Vec<Query> {
    let mut r = Rng(seed);
    (0..n).map(|_| {
        let d = r.next(); // hidden difficulty
        let p_small = 0.97 - 0.85 * d;
        let small_ok = r.next() < p_small;
        let large_ok = r.next() < 0.97 - 0.10 * d;
        // confidence: informative about p_small and the outcome, noisy
        let conf = (0.6 * p_small + 0.3 * (small_ok as u8 as f64) + 0.1 * r.next()).clamp(0.0, 1.0);
        Query { small_ok, large_ok, conf }
    }).collect()
}

struct Point { t: f64, acc: f64, cost: f64, esc: f64 } // per-query averages

/// Always call small; if conf < t also call large (the small call is wasted).
fn run(qs: &[Query], t: f64) -> Point {
    let (mut ok, mut cost, mut esc) = (0usize, 0.0, 0usize);
    for q in qs {
        cost += C_SMALL;
        if q.conf < t { cost += C_LARGE; esc += 1; ok += q.large_ok as usize; }
        else { ok += q.small_ok as usize; }
    }
    let n = qs.len() as f64;
    Point { t, acc: ok as f64 / n, cost: cost / n, esc: esc as f64 / n }
}

fn main() {
    let val = make_queries(20_000, 1);  // held-out set used to choose t
    let test = make_queries(20_000, 2); // fresh data to check the choice
    let curve: Vec<Point> = (0..=20).map(|i| run(&val, i as f64 / 20.0)).collect();
    for p in &curve { println!("{:.2} {:.3} {:5.2} {:5.1}%", p.t, p.acc, p.cost, 100.0 * p.esc); }

    let always_large = run(&val, 1.01).acc; // conf < 1.01 always: every query escalates
    let target = always_large - 0.02;       // allow 2 points of accuracy loss
    let best = curve.iter().find(|p| p.acc >= target).expect("target reachable");
    let on_test = run(&test, best.t);
    println!("t = {:.2}: val acc {:.3} cost {:.2}; test acc {:.3} cost {:.2} vs always-large {:.2}",
        best.t, best.acc, best.cost, on_test.acc, on_test.cost, C_LARGE);

    for w in curve.windows(2) { assert!(w[1].cost >= w[0].cost && w[1].esc >= w[0].esc); }
    assert!(best.acc >= target && best.cost < C_LARGE && on_test.cost < C_LARGE);
    assert!(on_test.acc > target - 0.01);
}

Run on an Apple M4 Pro (asserts pass): the sweep picks t = 0.50, escalating 44.8% of queries; validation accuracy 0.916 at 9.97 cost units, test 0.921 at 9.91, against 20 for always-large (accuracy 0.919). Accuracy peaks near t = 0.65 (0.941), above always-large: an artefact of the synthetic data, where the large model fails independently on easy queries; real errors correlate. No timing was measured.

Complexity and constants

• Expected cost per request: c_s + e * c_l (cascade, small call wasted on escalations) against c_l; it wins when e < 1 - c_s/c_l. At a 20x price ratio the break-even escalation rate is 95%, so the cost is dominated by e, not the ratio.
• Expected latency: L_s + e * L_l; an escalated request pays both calls in sequence. A classifier router adds one fixed cheap step and never pays twice.
• With an uninformative signal the accuracy–cost curve is the straight line between the two models; the better the signal, the more it bows above that line.

Efficiency

• Cost: per request, a router adds a rule lookup (ns), a small classifier pass (order of ms on CPU), or a second model call (a cascade escalation costs a full small call plus a large one). The bound is the large model's price and decode latency, paid on the escalated share only.
• Levers:
  • Rules before models — no router call for obvious classes — when task types are known up front.
  • Learned router instead of cascade — no wasted small call, no double latency (large-call savings reported: over 2x RouteLLM, up to 40% Hybrid LLM; their abstracts) — when quality labels exist for training.
  • Free signals (structured-output validity, tests) before costly ones (judge, k samples) — a judge call can cost as much as the small answer — when the free signal separates well.
  • Per-task-type thresholds — each segment gets its own accuracy–cost point — when types differ in difficulty.
  • Cache first (semantic cache, prompt caching) — removes the call — when requests repeat.
• Naive → standard → state of the art:
  • Always the largest model · cost c_l per request.
  • Rules plus a cascade with a tuned threshold · c_s + e*c_l, extra latency on escalations.
  • Learned routers and learned chains (RouteLLM, FrugalGPT) · reported up to 98% (FrugalGPT) and over 2x (RouteLLM) cost reduction at matched quality on their benchmarks.

Implementation notes

• Signals. Mean token log-prob is cheap but often miscalibrated; calibrate it on held-out labels before thresholding. A judge or verifier is stronger but adds a call. Agreement of k sampled answers costs k small calls. Task checks (valid JSON, schema, code tests, citations present) are free and exact; use them first.
• Cache interplay. A router that moves a conversation between models or replicas loses prompt-cache hits: the cache is per model and often per replica, and a small-to-large escalation re-prefills the whole prompt. Pin a session to one tier or route by prefix hash; escalate at turn boundaries, not mid-turn.
• Fallbacks. Treat rate limits and overloads as routing events: retry with backoff, cool down after repeated failures, then fail over (LiteLLM fallbacks, allowed_fails, cooldown_time; OpenRouter models list). Use client-side token buckets so the router does not cause the 429s.
• Drift. Provider model updates change both quality and calibration: keep a rolling audit sample sent to the large model, track escalation rate and regret, and refit when they move.

Pitfalls

• Confidence is not correctness: a small model can be confidently wrong, and the cascade silently ships it. Audit with sampled large-model answers.
• Tuning and testing on the same queries overstates savings; counting only the small model's price ignores that escalations pay both calls.
• Cascading when a tool call or side effect already happened: escalating after a non-idempotent tool call repeats it.
• Retry chains across providers multiply latency; cap attempts and total time.
• Router and models drift apart: a router trained on one model pair can degrade when either side is replaced, though RouteLLM reports transfer to new pairs without retraining on its benchmarks.

Variants and alternatives

In production

• Customer support: FAQ intents to a small model, disputes to a large one; escalate on low confidence.
• Coding assistants: completion on a small model, multi-file edits on a large one.
• Search/QA: small model over retrieved passages; escalate when citations are missing.
• Finance and legal extraction: small model fills fields, the large one redoes failed schema checks.
• Healthcare triage drafting: cheap draft, large model for complex cases, human review either way.
• Game NPC dialogue: on-device small model for chatter, cloud model for plot-critical scenes.
• Gateways: LiteLLM Router, OpenRouter fallbacks, RouteLLM server.

Implementations

Sources

• Chen, Zaharia, Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, 2023. paper
• Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, 2024. paper · code
• Ding et al., Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, 2024. paper
• Hu et al., RouterBench: A Benchmark for Multi-LLM Routing System, 2024. paper
• Viola, Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR 2001. pdf
• LiteLLM docs, Router: routing, fallbacks, cooldowns. docs
• OpenRouter docs, Model fallbacks. docs