Caches and queues

Deep dive

Service map

Delivery semantics

Decision table

Code

Cache-aside with stampede protection (Rust, redis crate):

use redis::{cmd, Connection, RedisResult};
use std::time::{SystemTime, UNIX_EPOCH};

/// Cache-aside with a per-key refill lock and jittered TTL (redis crate, raw commands).
fn get_user(con: &mut Connection, uid: u64, load: impl Fn(u64) -> String, ttl: u64) -> RedisResult<String> {
    let key = format!("user:{uid}");
    if let Some(hit) = cmd("GET").arg(&key).query::<Option<String>>(con)? {
        return Ok(hit);
    }
    // SET lock:key 1 NX EX 10 -> only one caller per key refills the cache
    let lock = format!("lock:{key}");
    let won = cmd("SET").arg(&lock).arg(1).arg("NX").arg("EX").arg(10).query::<Option<String>>(con)?;
    let val = load(uid);                                 // losers: better to sleep 10-50 ms and re-read
    if won.is_some() {
        let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().subsec_nanos() as u64;
        let jitter = nanos % (ttl / 10 + 1);             // spread expiries of keys written together
        cmd("SET").arg(&key).arg(&val).arg("EX").arg(ttl + jitter).query::<()>(con)?;
        cmd("DEL").arg(&lock).query::<()>(con)?;
    }
    Ok(val)
}

Postgres as a job queue:

CREATE TABLE jobs (id bigserial PRIMARY KEY, kind text, payload jsonb,
                   run_at timestamptz DEFAULT now(), attempts int DEFAULT 0);
CREATE INDEX ON jobs (run_at);
-- worker loop: claim one job without blocking other workers
WITH j AS (SELECT id FROM jobs WHERE run_at <= now()
           ORDER BY run_at LIMIT 1 FOR UPDATE SKIP LOCKED)
UPDATE jobs SET attempts = attempts + 1, run_at = now() + interval '5 minutes'
FROM j WHERE jobs.id = j.id RETURNING jobs.*;
-- on success: DELETE FROM jobs WHERE id = $1;

SQS consumer settings that prevent duplicate storms:

aws sqs set-queue-attributes --queue-url "$Q" --attributes '{
  "VisibilityTimeout": "120", "ReceiveMessageWaitTimeSeconds": "20",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"5\"}" }'

Efficiency

• Cost: a cache read is one network round trip plus microseconds of server work, so the round trip bounds a naive client: at 250 ms RTT a server that handles 100k requests/s serves at most four requests/s per connection (Redis pipelining docs). SQS bills per request, and batch actions handle up to 10 messages per call (SQS Developer Guide, as of 2026-10). Bound: latency and request count.
• Levers:
  • pipelining — many commands per round trip, and more operations per second on the server (Redis pipelining docs) — bulk reads and writes from one client
  • SQS batch actions — up to 10 messages per call, fewer billed requests (SQS batch actions doc) — producers and consumers with a backlog
  • SQS long polling, wait up to 20 s — fewer empty receives (SQS polling doc) — low-traffic queues
  • jittered TTL plus a per-key refill lock — no synchronized expiry stampede (Code above) — hot keys
  • admission policy such as W-TinyLFU — higher hit rate at equal memory (order of; see that entry) — skewed key popularity
• Naive → standard → state of the art:
  • one command or message per call, short polling · one RTT and one billed request per item
  • pipelined commands, batches of 10, long polling · RTTs and requests divided by the batch size
  • admission-aware cache plus request coalescing · one origin load per missed key

Implementation notes

• Make every consumer idempotent (dedupe key in the DB); at-least-once is the norm everywhere.
• Visibility timeout must exceed worst-case processing time, or messages are delivered twice concurrently.
• Long polling (20 s) on SQS cuts empty-receive cost by orders of magnitude.
• Retry with Exponential backoff with jitter; send poison messages to a DLQ after N attempts.
• Size caches with a hit-rate target; eviction policy matters (W-TinyLFU style admission beats plain LRU on skewed loads).

Pitfalls

• Thundering herd: a popular key expires and 10k requests hit the DB at once. Use locks, early refresh or request coalescing.
• Redis KEYS * or large HGETALL block the single-threaded server for seconds.
• Kafka cross-AZ replication: replication factor 3 over 3 zones pays inter-zone transfer twice per byte produced.
• FIFO queues cap throughput per message group; one hot group serializes everything.
• Caching errors or empty results without a short TTL hides recoveries.

Implementations

Sources

• Rajesh Nishtala et al., Scaling Memcache at Facebook, NSDI 2013. paper
• Jay Kreps, Neha Narkhede, Jun Rao, Kafka: a Distributed Messaging System for Log Processing, NetDB 2011.
• AWS, Amazon SQS Developer Guide (documentation). docs · Amazon ElastiCache pricing. pricing
• Redis documentation, Pipelining. docs