Presence tracking

Deep dive

How it works

Parts:

• connection table — conn → (user, expiry), one hash entry per socket
• user table — user → connection count, offline_at
• timer heap — earliest expiry first, re-armed lazily
• status publisher — emits Online/Offline transitions to subscribers (pub/sub fan-out)

Operations: connect, heartbeat, expiry sweep, disconnect with grace (steps below).

1. Gateway accepts a socket for user U: connect(U, conn). On the 0 to 1 transition emit Online(U).
2. Client (or the gateway, via protocol pings) heartbeats every H seconds; any inbound traffic also counts. Set expiry = now + TTL, with TTL of about 2-3 H so one lost packet is tolerated.
3. A timer heap fires at the earliest expiry. If the connection heartbeated meanwhile, re-arm at its new expiry; otherwise delete it.
4. When a user's last connection ends (clean close or expiry), do not emit yet: set offline_at = now + grace.
5. A new connection before offline_at cancels it silently. Otherwise emit Offline(U) and drop the user.
6. Transitions are published to subscribers (pub/sub fan-out); new subscribers get a snapshot first, then diffs.

states: Offline ─connect─► Online ─last conn gone─► Grace(offline_at) ─timeout─► Offline
                              ▲                         │ connect inside grace
                              └─────────────────────────┘

conn A  |--hb--hb--hb--x(crash)      TTL expires --> count 1 -> 0
conn B  |            |--hb--hb--...  connects inside grace
user    Online ------------------------------------ (no Offline emitted, no flap)

Across nodes, shard users by consistent hashing so one owner sees all of a user's connections; gateways forward connect, disconnect and aggregated heartbeats. If a gateway dies, expire all its connections together (gateway-level TTL, see phi accrual).

Code

use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};

pub type Ms = u64; // monotonic milliseconds
pub type UserId = u64;
pub type ConnId = u64;

#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum Event { Online(UserId), Offline(UserId) }

struct Conn { user: UserId, expiry: Ms }

#[derive(Default)]
struct User { conns: u32, offline_at: Option<Ms> }

pub struct Presence {
    ttl: Ms,   // a connection is dead if silent this long (about 2-3 heartbeats)
    grace: Ms, // the user stays "online" this long after the last connection died
    conns: HashMap<ConnId, Conn>,
    users: HashMap<UserId, User>,
    conn_timers: BinaryHeap<Reverse<(Ms, ConnId)>>,  // one entry per live connection
    grace_timers: BinaryHeap<Reverse<(Ms, UserId)>>, // lazy: stale entries skipped
}

impl Presence {
    pub fn new(ttl: Ms, grace: Ms) -> Self {
        Presence { ttl, grace, conns: HashMap::new(), users: HashMap::new(),
                   conn_timers: BinaryHeap::new(), grace_timers: BinaryHeap::new() }
    }

    /// New socket for `user`. Emits Online only on the 0 -> 1 transition.
    pub fn connect(&mut self, user: UserId, conn: ConnId, now: Ms, out: &mut Vec<Event>) {
        let expiry = now + self.ttl;
        self.conns.insert(conn, Conn { user, expiry });
        self.conn_timers.push(Reverse((expiry, conn)));
        let u = self.users.entry(user).or_default();
        let was_online = u.conns > 0 || u.offline_at.is_some(); // in grace = still online
        u.conns += 1;
        u.offline_at = None; // cancels a pending Offline: a quick reconnect is invisible
        if !was_online { out.push(Event::Online(user)); }
    }

    /// Heartbeat: one hash write, no heap work (the timer re-arms itself lazily).
    pub fn heartbeat(&mut self, conn: ConnId, now: Ms) -> bool {
        match self.conns.get_mut(&conn) {
            Some(c) => { c.expiry = now + self.ttl; true }
            None => false, // unknown or already expired: client must reconnect
        }
    }

    pub fn disconnect(&mut self, conn: ConnId, now: Ms) {
        if let Some(c) = self.conns.remove(&conn) { self.drop_conn(c.user, now); }
    }

    fn drop_conn(&mut self, user: UserId, now: Ms) {
        let u = self.users.get_mut(&user).unwrap();
        u.conns -= 1;
        if u.conns == 0 {
            let at = now + self.grace;
            u.offline_at = Some(at);
            self.grace_timers.push(Reverse((at, user)));
        }
    }

    pub fn is_online(&self, user: UserId) -> bool { self.users.contains_key(&user) }

    /// Call from a timer tick (e.g. every 100-1000 ms). Cost is proportional to timers that fire.
    pub fn sweep(&mut self, now: Ms, out: &mut Vec<Event>) {
        while let Some(&Reverse((t, conn))) = self.conn_timers.peek() {
            if t > now { break; }
            self.conn_timers.pop();
            match self.conns.get(&conn) {
                None => {} // disconnected earlier
                Some(c) if c.expiry > now => { // heartbeat arrived: re-arm at the new expiry
                    self.conn_timers.push(Reverse((c.expiry, conn)));
                }
                Some(c) => { let user = c.user; self.conns.remove(&conn); self.drop_conn(user, now); }
            }
        }
        while let Some(&Reverse((t, user))) = self.grace_timers.peek() {
            if t > now { break; }
            self.grace_timers.pop();
            if let Some(u) = self.users.get(&user) {
                if u.conns == 0 && u.offline_at == Some(t) { // not cancelled or re-armed
                    self.users.remove(&user);
                    out.push(Event::Offline(user));
                }
            }
        }
    }
}

Tested with asserts: two devices give one Online; expiry then grace then one Offline; a disconnect and reconnect inside grace emits nothing; a heartbeat for an expired connection returns false.

Complexity and constants

• Heartbeat: one hash lookup and write, no allocation, no heap work. Measured here (Apple silicon, single thread, 1M connections, random access): about 30 ns per heartbeat, about 75 ns per connect, expiring 1M connections about 250 ms in one sweep.
• Memory: order of 100 bytes per connection and per user in this layout (two hash entries plus one heap entry); roughly 100 MB per million connections.
• Inbound load = connections / H: 1M connections at H = 15 s is order of 67k heartbeats/s.
• Detection latency for a silent death: up to TTL, then grace. Phoenix Tracker defaults: 15 s heartbeat, 30 s until a silent replica is flagged down, 20 min until it is permanently removed.

Efficiency

• Cost: a heartbeat is one hash lookup and write, about 30 ns; connect about 75 ns; sweeping 1M expiries about 250 ms (measured above, Apple silicon, single thread). Memory order of 100 bytes per connection, about 100 MB per million. Inbound load is connections / H: 1M connections at H = 15 s is order of 67k heartbeats/s. Bound: network I/O and fan-out of status changes, not CPU.
• Levers:
  • piggyback liveness on any frame, heartbeat only after an idle H — chatty clients cost nothing extra — interactive connections
  • WebSocket ping/pong — a few bytes, handled before application code (RFC 6455) — WebSocket gateways
  • lazy timer re-arm — heap work per expiry, not per heartbeat — many connections
  • grace period — a reconnect costs zero events instead of two times every watcher — mobile and flaky links
  • subscribe only for users on screen, batch gateway heartbeats — fan-out bounded by viewers; one message per gateway per interval — large friend or device lists
  • last_seen timestamp instead of live status — no fan-out at all — coarse presence
• Naive → standard → state of the art:
  • heartbeat per socket, event on every flap · one event per transition times every watcher
  • TTL plus grace plus lazy timers · tens of ns per heartbeat
  • gateway-batched heartbeats, on-screen subscriptions, delta-CRDT replication (Phoenix.Tracker) · fan-out bounded by viewers

Implementation notes

• Use a monotonic clock for every deadline; never mix in wall time.
• Tick the sweep on a fixed timer (100 ms to 1 s) rather than one OS timer per connection; one tick drains every expired entry.
• Redis variant: SET presence:{user}:{conn} 1 EX ttl per connection, EXPIRE to refresh. Expiry events (keyspace notifications) are generated when Redis actually deletes the key, which can lag the TTL, and Pub/Sub is fire and forget, so a missed event is lost: reconcile periodically by scanning, and do not make correctness depend on the event.
• Add jitter to heartbeat timers, so a restart does not synchronize 1M clients into one burst.
• Include a connection epoch or id in connect so a late disconnect of an old socket cannot kill a newer one.
• On owner failover, start every user with a fresh TTL and let heartbeats reconfirm; do not emit Offline for users you simply have not heard from yet.

Pitfalls

• Counting users, not connections: closing one of two tabs marks the user offline.
• Emitting Offline immediately on socket close: mobile networks flap, so every watcher sees a storm of toggles.
• TTL equal to H: one delayed heartbeat flaps the user. Use at least 2 H.
• Fan-out to all friends on every change: celebrity accounts multiply it into millions of messages. Cap, sample, or require a subscription.
• Presence state in a single Redis key per user with a TTL: multi-device users overwrite each other. Track per-connection entries.
• Heartbeats stuck behind a slow handler on the same thread; run liveness on its own path.
• Relying on exact "now": presence is eventually consistent; never use it to gate a write that needs safety.

Variants and alternatives

In production

Messaging and collaboration: Phoenix Presence on Phoenix.Tracker (Elixir, CRDT-based, per-topic), Centrifugo online presence per channel (Redis or built-in engines), Ably and Pusher presence channels, Socket.IO rooms (application-built), XMPP servers (ejabberd, Prosody) for roster presence. IoT: MQTT brokers (EMQX) deliver a client's Will message when its connection dies without a clean close. Infrastructure: Kubernetes kubelets heartbeat to a per-node Lease object and the control plane judges node availability from its renewTime.

Implementations

Sources

• Paulo Sérgio Almeida, Ali Shoker, Carlos Baquero, Delta State Replicated Data Types, J. Parallel Distrib. Comput. 2018. paper
• Phoenix.Tracker documentation (heartbeat, down and permdown defaults). docs
• Phoenix.Presence documentation. docs
• Centrifugo, Online presence. docs
• Redis, Keyspace notifications (timing of expired events, fire-and-forget Pub/Sub). docs
• P. Saint-Andre, XMPP: Instant Messaging and Presence, RFC 6121, 2011. rfc
• OASIS, MQTT Version 3.1.1 (Will Message). spec
• Kubernetes documentation, Leases (node heartbeats). docs