Memory consolidation

Deep dive

How it works

Parts:

• memory record — text, unit vector, importance, last_used, strength
• vector index — nearest-neighbour lookup for the write path and clustering
• decision model — LLM call choosing ADD, UPDATE, DELETE or NOOP, and writing merged texts
• scorer — importance × exp(-(now − last_used) / (strength · τ))
• offline job — merge, decay, drop, cap

Operations:

1. Extract candidate facts from a turn (one model call): short, self-contained statements, each with an importance rating (e.g. 1-10).
2. Write path: embed the fact, fetch the nearest s memories, let a model pick ADD (nothing equivalent), UPDATE (augment or replace a memory), DELETE (contradicted), NOOP (already known).
3. Recall bookkeeping: when a memory is retrieved and used, set last_used = now and strength += 1.
4. Score each memory as importance × exp(-(now − last_used) / (strength · τ)). Strong, recent, important memories survive.
5. Offline pass (idle, or when summed importance of new memories passes a threshold): merge pairs above a similarity threshold into one text and a strength-weighted centroid, drop memories below a floor, keep the top cap.
6. Optional reflection: summarize clusters into higher-level facts and store them as new memories that link their sources.

new fact ─embed─► top-s neighbours ─LLM─► ADD | UPDATE | DELETE | NOOP
                                              │
store ──(idle)──► cluster by cosine ─► merge text ─► decay ─► drop weak ─► cap

Code

#[derive(Clone, Debug)]
pub struct Mem {
    pub text: String,
    pub vec: Vec<f32>,   // unit length, so cosine = dot product
    pub importance: f32, // 0..1, rated once at write time
    pub last_used: f64,  // hours; reset on every recall
    pub strength: f32,   // +1 per recall or merge: slows forgetting
}

pub enum Op { Add, Update(usize, String), Delete(usize), Noop } // Mem0-style write decision

pub struct Store { pub mems: Vec<Mem>, pub tau_h: f64 } // tau_h: base forgetting time constant, hours

fn dot(a: &[f32], b: &[f32]) -> f32 { a.iter().zip(b).map(|(x, y)| x * y).sum() }
fn unit(mut v: Vec<f32>) -> Vec<f32> {
    let n = dot(&v, &v).sqrt().max(1e-12);
    v.iter_mut().for_each(|x| *x /= n);
    v
}

impl Store {
    /// Ebbinghaus-style retention exp(-t / (strength * tau)), weighted by importance.
    pub fn score(&self, m: &Mem, now: f64) -> f32 {
        let t = (now - m.last_used).max(0.0);
        m.importance * (-(t / (m.strength as f64 * self.tau_h)) as f32).exp()
    }

    /// Write path: show the k nearest memories to `decide` (an LLM call in practice).
    pub fn ingest(&mut self, text: &str, vec: Vec<f32>, importance: f32, now: f64, k: usize,
                  decide: impl Fn(&str, &[(f32, &Mem)]) -> Op) {
        let v = unit(vec);
        let mut near: Vec<(f32, usize)> = self.mems.iter().enumerate().map(|(i, m)| (dot(&v, &m.vec), i)).collect();
        near.sort_by(|a, b| b.0.total_cmp(&a.0));
        near.truncate(k);
        let view: Vec<(f32, &Mem)> = near.iter().map(|&(s, i)| (s, &self.mems[i])).collect();
        match decide(text, &view) {
            Op::Add => self.mems.push(Mem { text: text.into(), vec: v, importance, last_used: now, strength: 1.0 }),
            Op::Update(i, t) => { let m = &mut self.mems[i]; m.text = t; m.last_used = now; m.strength += 1.0; }
            Op::Delete(i) => { self.mems.swap_remove(i); }
            Op::Noop => {}
        }
    }

    /// Offline pass ("sleep"): merge near-duplicates, forget the weak, enforce capacity.
    /// `merge` stands in for an LLM call that writes one text from two. Returns (merged, dropped).
    pub fn consolidate(&mut self, now: f64, sim: f32, floor: f32, cap: usize,
                       merge: impl Fn(&str, &str) -> String) -> (usize, usize) {
        let (mut merged, mut alive) = (0, vec![true; self.mems.len()]);
        for i in 0..self.mems.len() {
            if !alive[i] { continue; }
            for j in i + 1..self.mems.len() {
                if !alive[j] || dot(&self.mems[i].vec, &self.mems[j].vec) < sim { continue; }
                let b = self.mems[j].clone();
                let a = &mut self.mems[i];
                let v: Vec<f32> = a.vec.iter().zip(&b.vec).map(|(x, y)| x * a.strength + y * b.strength).collect();
                a.vec = unit(v); // strength-weighted centroid
                a.text = merge(&a.text, &b.text);
                a.importance = a.importance.max(b.importance);
                a.last_used = a.last_used.max(b.last_used);
                a.strength += b.strength;
                alive[j] = false;
                merged += 1;
            }
        }
        let mut keep: Vec<Mem> = self.mems.drain(..).zip(alive).filter(|(_, a)| *a).map(|(m, _)| m).collect();
        let before = keep.len();
        keep.retain(|m| self.score(m, now) >= floor);
        keep.sort_by(|a, b| self.score(b, now).total_cmp(&self.score(a, now)));
        keep.truncate(cap);
        let dropped = before - keep.len();
        self.mems = keep;
        (merged, dropped)
    }
}

#[test]
fn consolidates() {
    let mk = |t: &str, v: [f32; 2], imp, last| Mem { text: t.into(), vec: unit(v.to_vec()), importance: imp, last_used: last, strength: 1.0 };
    let mut s = Store { tau_h: 72.0, mems: vec![
        mk("uses vim", [1.0, 0.0], 0.6, 100.0),
        mk("prefers vim keybindings", [0.99, 0.1], 0.8, 90.0), // near-duplicate
        mk("lunch was soup", [0.0, 1.0], 0.1, 0.0),            // old and unimportant
    ]};
    let (merged, dropped) = s.consolidate(200.0, 0.95, 0.05, 10, |a, b| format!("{a}; {b}"));
    assert_eq!((merged, dropped, s.mems.len()), (1, 1, 1));
    assert_eq!((s.mems[0].strength, s.mems[0].importance), (2.0, 0.8));
}

decide and merge are closures so tests can stub them; in production each is one model call. Vectors are normalized once, so similarity is a dot product.

Complexity and constants

• Write path: one embedding, one nearest-neighbour query (O(log n) with HNSW), one decision call over the retrieved texts.
• Brute-force offline pass: n²/2 dot products of d floats. For n = 10,000 and d = 1,024 that is 5·10⁷ pairs and about 5·10¹⁰ multiply-adds (arithmetic, not a benchmark); with an index, n queries of k neighbours each.
• Storage: 4·d bytes per vector in f32, 4 KB at d = 1,024, so 10,000 memories take about 40 MB plus text; int8 vectors are 4x smaller.
• Cost drivers: LLM calls (extract per turn, decide per fact, merge per cluster) dominate dollars; the vector math is cheap.

Efficiency

• Cost: a write is one embedding, one nearest-neighbour query and one decision call; the offline pass is one merge call per cluster. Model calls and tokens dominate; vector math is cheap (5·10¹⁰ multiply-adds for the brute-force pass at n = 10,000, d = 1,024, arithmetic only). Memory is 4·d bytes per f32 vector, 4 KB at d = 1,024. Bound: LLM latency and token cost.
• Levers:
  • structured memory instead of full context — 91% lower p95 latency and over 90% fewer tokens on LOCOMO (Mem0 paper) — long multi-session dialogues
  • sleep-time work — about 5x less test-time compute at equal accuracy on Stateful GSM-Symbolic and AIME (Lin et al. 2025) — idle periods, predictable queries
  • similarity filter before the model — only pairs above the threshold reach a call; exact duplicates merge by hash — any pass
  • one call per cluster, one call per turn's facts — fewer calls — batched merges
  • index instead of brute force — O(n·k) instead of O(n²·d) — stores beyond about 10⁴ items
• Naive → standard → state of the art:
  • append everything, retrieve top-k · prompt and search grow without bound
  • per-fact ADD/UPDATE/DELETE/NOOP plus capped decay · one decision call per fact
  • the above plus background sleep-time agents that rewrite memory off the critical path · calls leave the user's latency path

Implementation notes

• Run the pass over a snapshot and swap atomically; keep tombstones (soft delete) for a grace period so a bad merge is reversible.
• Calibrate the merge threshold per embedding model on labelled duplicate pairs; the right cosine differs per model. Negation and changed numbers stay close in embedding space, so route borderline pairs to the model rather than auto-merging.
• Keep source ids on each merged memory for provenance and for cluster re-splitting.
• For temporal facts ("lives in Berlin"), store a validity interval and close it instead of deleting; temporal knowledge-graph memories (Zep, Graphiti) do this.
• Store the importance score at write time; re-rating on every pass multiplies model cost.
• Keep a hard cap on the memory text injected per turn, whatever the store size.
• Exact duplicates can be removed cheaply with a hash of normalized text; for long memories use MinHash.

Pitfalls

• Drift: merging merged texts repeatedly loses detail; keep the source facts or ids and regenerate from them.
• Rich get richer: recall raises strength, so early memories dominate; cap strength or add a small exploration bonus.
• Forgetting the rare critical fact (allergy, deploy credential location): importance must be able to pin a memory; never decay user-pinned items.
• Merging contradictions into mush: "uses vim; uses emacs". Detect conflicts and keep the newer fact with a pointer to the old one.
• Poisoned memory: text from tool output or web pages that gets stored persists across sessions and is later trusted; only store facts derived from the user or verified results, see prompt-injection defense.
• Cluster chaining: single-link merging joins A-B and B-C although A and C differ; merge against a centroid with a threshold, as in the code, and re-check.
• Running consolidation during a live turn adds latency and races with writes.

Variants and alternatives

In production

Mem0 runs the extract-then-ADD/UPDATE/DELETE/NOOP pipeline; Letta pairs pinned memory blocks with background "sleep-time" agents that rewrite memory; LangMem ships a background memory manager that extracts, consolidates and updates memories; Zep and Graphiti keep time-stamped facts and invalidate superseded ones.

Implementations

Sources

• Joon Sung Park et al., Generative Agents: Interactive Simulacra of Human Behavior, UIST 2023. paper
• Prateek Chhikara et al., Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory, 2025. paper · code
• Wanjun Zhong et al., MemoryBank: Enhancing Large Language Models with Long-Term Memory, AAAI 2024. paper · code
• Kevin Lin et al., Sleep-time Compute: Beyond Inference Scaling at Test-time, 2025. paper
• Preston Rasmussen et al., Zep: A Temporal Knowledge Graph Architecture for Agent Memory, 2025. paper · code
• Charles Packer et al., MemGPT: Towards LLMs as Operating Systems, 2023. paper