Deep ranking models for recommendation

Deep dive

How it works

Parts:

• embedding tables — one row per hashed ID bucket, per sparse feature (feature hashing); most of the bytes
• dense tower — numeric features (age of item, counts, ratios) through an MLP into the same width
• interaction layer — pairwise dot products (DLRM), cross layers (DCN, DCN-V2) or a wide linear part (Wide & Deep)
• sequence module — target attention over the user's recent events, weighted by the candidate (DIN) or a transformer over the history
• shared trunk — experts or shared layers; MMoE gives each task its own softmax gate over the experts
• heads — one sigmoid per feedback type (click, like, dwell > t, share, hide)
• value combiner — score = Σ wₖ·pₖ, with negative weights for hide or report
• feature service — user and item features at request time (feature store)

1. Retrieve candidates (two-tower, social graph, popularity) and attach features: user side once per request, item side per candidate.
2. Look up one embedding row per sparse feature; pool multi-valued features.
3. Interact. DLRM stacks the dense vector and all embeddings and takes every pairwise dot product; DCN-V2 uses x_{l+1} = x₀ ⊙ (W x_l + b) + x_l (details).
4. Attend over history (DIN): u = Σᵢ a(eᵢ, e_cand)·eᵢ with a small net a, so the same history yields a different interest vector for every candidate. This makes the user side per-candidate again.
5. Predict every head from the trunk (MMoE: task_k = headₖ(Σᵢ softmax(Gₖx)ᵢ · expertᵢ(x))).
6. Score with Σ wₖ·pₖ, take the top n (top-k selection); a re-ranker adds diversity.
7. Train on logged impressions: one label per head, cross-entropy summed over heads; downsample negatives (negative sampling) and correct the odds afterwards.

 sparse ids ─hash─► embedding rows ─┐
 history ──► target attention ──────┤
 dense ──► dense MLP ───────────────┴─► interaction ─► shared trunk/experts
                                                           ├─ head click ─► p1 ┐
                                                           ├─ head like  ─► p2 ├─► Σ wk·pk ─► top-n
                                                           └─ head hide  ─► p3 ┘

Design points

• Value weights. They encode what the product values (share above click, hide negative), are tuned in online experiments, and live in config. Calibrated heads keep them meaningful across retrains.
• Position bias. Top slots get clicks regardless of quality. Common practice: train with display position as a feature, serve with one default value.
• Exploration. A greedy ranker never learns about items it does not show; reserve a traffic slice for bandit exploration of new items.
• Multi-task. Weakly related tasks hurt each other in a fully shared trunk; MMoE lets each task weight the experts itself (Ma et al. 2018).
• Watch time. Covington et al. (YouTube) train logistic regression with positives weighted by watch time, to predict expected watch time.

Code

DLRM-shaped forward pass (two hashed tables, dense layer, pairwise dots, click and like heads, weighted value) with SGD on the heads and the two touched rows; the dense layer is frozen to keep the backward pass short.

const D: usize = 8; // embedding width
const ND: usize = 4; // dense features
const NB: usize = 1 << 10; // hash buckets per table
const NP: usize = 3; // pairs among 3 vectors (dense, user, topic)
const IN: usize = NP + D; // head input: pair dots ++ dense vector

fn hash(id: u64, salt: u64) -> usize { // multiplicative hash into the table
    ((id ^ salt).wrapping_mul(0x9E37_79B9_7F4A_7C15) >> 40) as usize % NB
}
fn sigmoid(z: f32) -> f32 { 1.0 / (1.0 + (-z).exp()) }
fn dot(a: &[f32], b: &[f32]) -> f32 { a.iter().zip(b).map(|(x, y)| x * y).sum() }

struct Rng(u64);
impl Rng {
    fn next(&mut self) -> f32 { // xorshift64, uniform in [-1, 1)
        self.0 ^= self.0 << 13; self.0 ^= self.0 >> 7; self.0 ^= self.0 << 17;
        (self.0 >> 40) as f32 / (1u64 << 23) as f32 - 1.0
    }
}

struct Model {
    user: Vec<f32>,         // NB x D hashed tables
    topic: Vec<f32>,
    bottom: [[f32; ND]; D], // dense MLP layer (frozen in this sketch)
    head: [[f32; IN + 1]; 2], // heads: click, like (weights + bias)
    value_w: [f32; 2],      // value = 1*p_click + 3*p_like
}

struct Fwd { v: [[f32; D]; 3], x: [f32; IN], p: [f32; 2] }

impl Model {
    fn new(r: &mut Rng) -> Self {
        let mut tab = || (0..NB * D).map(|_| 0.1 * r.next()).collect::<Vec<_>>();
        let (user, topic) = (tab(), tab());
        let mut bottom = [[0.0; ND]; D];
        for row in bottom.iter_mut() { for w in row.iter_mut() { *w = 0.5 * r.next(); } }
        Model { user, topic, bottom, head: [[0.0; IN + 1]; 2], value_w: [1.0, 3.0] }
    }
    fn forward(&self, u: u64, t: u64, dense: &[f32; ND]) -> Fwd {
        let (hu, ht) = (hash(u, 1) * D, hash(t, 2) * D);
        let mut v = [[0.0; D]; 3];
        for i in 0..D { v[0][i] = dot(&self.bottom[i], dense).max(0.0); }
        v[1].copy_from_slice(&self.user[hu..hu + D]);
        v[2].copy_from_slice(&self.topic[ht..ht + D]);
        let mut x = [0.0; IN];
        for (k, (i, j)) in [(0, 1), (0, 2), (1, 2)].into_iter().enumerate() {
            x[k] = dot(&v[i], &v[j]); // pairwise dot interaction
        }
        x[NP..].copy_from_slice(&v[0]);
        let p = [0, 1].map(|h| sigmoid(dot(&self.head[h][..IN], &x) + self.head[h][IN]));
        Fwd { v, x, p }
    }
    fn value(&self, f: &Fwd) -> f32 { dot(&self.value_w, &f.p) }
    fn sgd(&mut self, u: u64, t: u64, dense: &[f32; ND], y: [f32; 2], lr: f32) {
        let f = self.forward(u, t, dense);
        let mut dv = [[0.0f32; D]; 3];
        for h in 0..2 {
            let dz = f.p[h] - y[h]; // d logloss / d logit
            for k in 0..IN {
                let dx = dz * self.head[h][k];
                if k < NP { let (i, j) = [(0, 1), (0, 2), (1, 2)][k];
                    for d in 0..D { dv[i][d] += dx * f.v[j][d]; dv[j][d] += dx * f.v[i][d]; }
                } else { dv[0][k - NP] += dx; }
                self.head[h][k] -= lr * dz * f.x[k];
            }
            self.head[h][IN] -= lr * dz;
        }
        let (hu, ht) = (hash(u, 1) * D, hash(t, 2) * D); // sparse update: two rows only
        for d in 0..D {
            self.user[hu + d] -= lr * dv[1][d];
            self.topic[ht + d] -= lr * dv[2][d];
        }
    }
}

fn fav(u: u64) -> u64 { (u * 3) % 8 } // user u likes topic fav(u)

fn main() {
    let mut r = Rng(0x1234_5678_9ABC_DEF1);
    let mut m = Model::new(&mut r);
    let mut feats = |r: &mut Rng| [r.next(), r.next(), r.next(), r.next()];
    for _ in 0..60_000 { // logged impressions: clicks follow the favourite topic
        let (u, t) = ((r.next().abs() * 16.0) as u64 % 16, (r.next().abs() * 8.0) as u64 % 8);
        let dense = feats(&mut r);
        let click = (t == fav(u)) as u8 as f32;
        m.sgd(u, t, &dense, [click, click * (dense[0] > 0.0) as u8 as f32], 0.05);
    }
    let n = 1000;
    let cands: Vec<(u64, [f32; ND])> = (0..n).map(|i| (i % 8, feats(&mut r))).collect();
    let score = |u: u64| -> Vec<f32> { cands.iter().map(|(t, d)| m.value(&m.forward(u, *t, d))).collect() };
    let s = score(5);
    assert_eq!(s.len(), n as usize);
    assert_eq!(s, score(5)); // deterministic
    let mut order: Vec<usize> = (0..s.len()).collect();
    order.sort_by(|&a, &b| s[b].total_cmp(&s[a]));
    assert!(order[..10].iter().all(|&i| cands[i].0 == fav(5))); // history match ranks first
    let t0 = std::time::Instant::now();
    let reqs = 2000;
    let sink: f32 = (0..reqs).map(|q| score(q % 16)[0]).sum();
    let dt = t0.elapsed().as_secs_f64();
    println!("top value {:.2} (sink {sink:.1}); {:.1} M candidates/s", s[order[0]], (n * reqs) as f64 / dt / 1e6);
}

Run on an Apple M4 Pro (measured): the asserts pass (the ten best candidates of user 5 all have the user's favourite topic) and scoring ran at 50-55 M candidates/s on one thread.

Complexity and constants

• Per candidate: s sparse lookups of d floats (4·s·d bytes; 20 × 64 × 4 ≈ 5 KB, each row a likely cache miss), s(s+1)/2 dots of length d, one MLP; DIN adds L activation-net evaluations for history length L.
• Memory: rows × d × 4 bytes; 10⁹ rows at d = 64 ≈ 256 GB in fp32, so tables are sharded.
• Training updates only the touched rows (embeddings).

Efficiency

• Cost: the toy above (tables 32 KB each, cache-resident) measured about 18-20 ns per candidate on one M4 Pro core. That is a floor: real tables do not fit cache, so 20 random 256-byte row reads per candidate are bound by memory latency and bandwidth. Per request: candidates × per-candidate cost, plus the feature fetches.
• Levers:
  • compute user-side features and user-only layers once per request — saves their share of per-candidate work (arithmetic) — N ≫ 1 candidates
  • score candidates as one matrix batch — lets GEMM and GPU run efficiently (practice) — hundreds of candidates
  • fp16 or int8 table rows — 2-4× fewer bytes (arithmetic) — memory-bound tables (quantization); recheck calibration
  • cache hot rows, shard the rest — few rows take most lookups (heavy-tailed IDs) — embedding-table-sharding
  • cascade: a light model cuts N before the heavy one — cost ∝ N at each stage (practice) — N in the thousands
• Naive → standard → state of the art:
  • features fetched per candidate over the network, scored one by one · a round trip per candidate
  • batched candidates, user features once, sharded tables, shared multi-task trunk · lookup-bound
  • target attention over retrieved history events, quantized and cached rows, GPU serving · lookup + attention bound

Implementation notes

• Hash with a per-table salt; send rare IDs to one shared bucket.
• Row-major tables, d a multiple of 16 floats (one cache line); batch a request's lookups so misses overlap.
• Start heads at zero; set the bias to the log-odds of the positive rate.
• Log the features used at serving and train on those; recomputing them later causes train-serve skew.

Pitfalls

• Training on shown impressions only teaches the old ranker's taste; keep a randomised slice.
• Click-only value amplifies clickbait; add dwell, share and negative feedback.
• Uncalibrated heads: after a retrain the scale shifts and the effective weights move.
• Hash collisions on frequent IDs: give them their own rows.
• Late labels (dwell, purchase) counted as negatives bias probabilities down (CTR models).

Variants and alternatives

In production

• Video: YouTube's two-stage system, a deep candidate generator and a deep ranker (Covington et al.).
• App store: Wide & Deep is deployed in Google Play (paper).
• Display ads and e-commerce: DIN serves the main traffic of Alibaba's display advertising system (paper).
• Social feeds, explore pages, news, music and job boards use the same pattern (no per-company paper cited here).

Implementations

Sources

• Covington, Adams, Sargin, Deep Neural Networks for YouTube Recommendations, RecSys 2016. DOI
• Cheng et al., Wide & Deep Learning for Recommender Systems, DLRS 2016. arXiv
• Naumov et al., Deep Learning Recommendation Model for Personalization and Recommendation Systems, 2019. arXiv · code
• Wang et al., Deep & Cross Network for Ad Click Predictions, ADKDD 2017. DOI
• Wang et al., DCN V2, WWW 2021. arXiv
• Zhou et al., Deep Interest Network for Click-Through Rate Prediction, KDD 2018. arXiv
• Ma et al., Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts, KDD 2018. DOI