Webhook delivery

Deep dive

How it works

Parts:

• outbox — the event plus one delivery row per subscribed endpoint (attempts, next due time), written in the business transaction.
• delivery worker — claims due rows, signs at send time, POSTs, schedules the next attempt; about 8 attempts over 28 h, HMAC about 0.7 µs per KB.
• address guard — resolves the host, rejects non-public addresses, connects to that address only.
• endpoint health — a failure clock per endpoint; disables it after days of failure.
• receiver — verifies timestamp and signature, dedupes by event id, answers 2xx fast.

Sender:

1. In the same database transaction as the state change, insert the event and one delivery row per subscribed endpoint (transactional outbox). Filter by event type here, not at the receiver.
2. A worker claims due rows, builds headers webhook-id (constant across retries), webhook-timestamp (fresh per attempt, so sign at send time) and webhook-signature (space-separated, one entry per live key).
3. Before connecting: resolve the host, reject non-public addresses, connect to that address only.
4. POST with short connect and total timeouts; do not follow redirects.
5. Classify: 2xx done; 410 disable the endpoint; 429 or retry-after stretch the next delay; everything else (3xx, 4xx, 5xx, timeout, reset) is a failure. Schedule the next attempt from a table, plus jitter.
6. After days of consecutive failure disable the endpoint, tell the owner out of band, keep events replayable.

Receiver: read raw body; check timestamp; verify each signature in constant time; insert the event id under a unique constraint in the same transaction as the effect; answer 2xx; do slow work from a queue.

txn ─► event + delivery rows ─► worker ─sign─► guard ─POST─► endpoint
              ▲                   │ fail: next due = backoff + jitter       │
              └───────────────────┘ days failing: disable, notify owner     ▼
                                          receiver: verify ─► dedupe ─► 2xx ─► queue

Code

Signing and verification with hmac + sha2 (RustCrypto; no hand-rolled hash), replay-checked receiver, status classification, jittered schedule, endpoint disabling, SSRF address check.

use hmac::{Hmac, KeyInit, Mac};
use sha2::Sha256;
use std::collections::HashMap;
use std::net::IpAddr;

/// Standard Webhooks "v1": base64(HMAC-SHA256(key, "{id}.{ts}.{body}")), prefixed "v1,".
pub fn sign(key: &[u8], id: &str, ts: u64, body: &[u8]) -> String {
    let mut m = Hmac::<Sha256>::new_from_slice(key).expect("any key length");
    for part in [id.as_bytes(), b".", ts.to_string().as_bytes(), b".", body] {
        m.update(part);
    }
    format!("v1,{}", b64(&m.finalize().into_bytes()))
}

fn b64(d: &[u8]) -> String {
    const A: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
    d.chunks(3).flat_map(|c| {
        let n = (c[0] as u32) << 16 | (*c.get(1).unwrap_or(&0) as u32) << 8 | *c.get(2).unwrap_or(&0) as u32;
        (0..4).map(move |i| if i <= c.len() { A[(n >> (18 - 6 * i) & 63) as usize] as char } else { '=' })
    }).collect()
}

fn ct_eq(a: &[u8], b: &[u8]) -> bool { // no early exit on the first differing byte
    a.len() == b.len() && a.iter().zip(b).fold(0u8, |d, (x, y)| d | (x ^ y)) == 0
}

#[derive(Debug, PartialEq)]
pub enum Reject { Stale, BadSignature, Replay }

pub struct Receiver { pub keys: Vec<Vec<u8>>, pub tolerance: u64, seen: HashMap<String, u64> }

impl Receiver {
    pub fn new(keys: Vec<Vec<u8>>, tolerance: u64) -> Self { Receiver { keys, tolerance, seen: HashMap::new() } }

    /// `body` must be the raw bytes as received, never re-serialized JSON.
    pub fn accept(&mut self, id: &str, ts: &str, sigs: &str, body: &[u8], now: u64) -> Result<(), Reject> {
        let ts: u64 = ts.parse().map_err(|_| Reject::Stale)?;
        if now.abs_diff(ts) > self.tolerance { return Err(Reject::Stale); }
        // header = space-separated signatures; several keys are live during rotation
        let ok = self.keys.iter().any(|k| {
            let want = sign(k, id, ts, body);
            sigs.split(' ').any(|s| ct_eq(s.as_bytes(), want.as_bytes()))
        });
        if !ok { return Err(Reject::BadSignature); }
        let window = self.tolerance * 2;
        self.seen.retain(|_, t| now.saturating_sub(*t) <= window);
        // replay cache only; processing itself must still be idempotent per id (durable store)
        if self.seen.insert(id.to_string(), now).is_some() { return Err(Reject::Replay); }
        Ok(())
    }
}

#[derive(Debug, PartialEq)]
pub enum Outcome { Delivered, Retry { retry_after: Option<u64> }, DisableEndpoint }

pub fn classify(status: u16, retry_after: Option<u64>) -> Outcome {
    match status {
        200..=299 => Outcome::Delivered,
        410 => Outcome::DisableEndpoint,                       // receiver says "stop"
        _ => Outcome::Retry { retry_after },                   // 3xx, 4xx, 5xx, timeout: all failures
    }
}

/// Delay before attempt `n+1` (n = attempts made so far, >= 1), seconds. None = give up.
pub const SCHEDULE: [u64; 8] = [0, 5, 300, 1800, 7200, 18_000, 36_000, 36_000];
pub fn next_delay(n: usize, retry_after: Option<u64>, rng: &mut u64) -> Option<u64> {
    let base = *SCHEDULE.get(n)?;
    *rng ^= *rng << 13; *rng ^= *rng >> 7; *rng ^= *rng << 17;
    let d = base + *rng % (base / 5 + 1); // jitter: up to +20 %, never earlier than the schedule
    Some(d.max(retry_after.unwrap_or(0)))
}

#[derive(Default)]
pub struct Endpoint { first_fail: Option<u64>, pub disabled: bool }
impl Endpoint {
    pub fn record(&mut self, ok: bool, now: u64) {
        if ok { self.first_fail = None; return; }
        let t = *self.first_fail.get_or_insert(now);
        if now - t >= 5 * 86_400 { self.disabled = true; } // 5 days of failures, as Svix does
    }
}

/// SSRF guard: resolve once, check every address, connect to that address, never re-resolve.
pub fn is_public(ip: IpAddr) -> bool {
    match ip {
        IpAddr::V4(a) => {
            let o = a.octets();
            !(a.is_private() || a.is_loopback() || a.is_link_local() || a.is_unspecified()
                || a.is_broadcast() || a.is_documentation()
                || (o[0] == 100 && o[1] & 0xc0 == 64) // 100.64.0.0/10 carrier NAT
                || o[0] >= 240)                       // reserved
        }
        IpAddr::V6(a) => match a.to_ipv4_mapped() {
            Some(v4) => is_public(IpAddr::V4(v4)),
            None => { let s = a.segments();
                !(a.is_loopback() || a.is_unspecified() || s[0] & 0xfe00 == 0xfc00 || s[0] & 0xffc0 == 0xfe80) }
        },
    }
}

Compiled and tested (hmac 0.13, sha2 0.11): signature recomputed against Python's hmac for the Standard Webhooks library's test id, secret and body; stale, tampered, replayed and rotated-key cases; the schedule sums to 99,305 s before jitter; private, link-local, CGNAT and IPv4-mapped addresses rejected.

Complexity and constants

• Per attempt: one HMAC over body + ~50 B, one TCP+TLS setup if no connection is warm (1 RTT TCP, 1 RTT TLS 1.3), one request, one response. Retries multiply this by about 8 in the worst case.
• Signing is cheap: measured here, 0.7 us for 1 KB and about 2.5 GB/s on large bodies (single thread, Apple M4 Pro, release build). The cost drivers are round trips and the receiver's latency, not CPU.
• Storage: one row per (event, endpoint); body stored once. Svix's retry gaps: 5 s, 5 min, 30 min, 2 h, 5 h, 10 h, 10 h.

Efficiency

• Cost: signing is cheap: 0.7 µs for 1 KB, about 2.5 GB/s on large bodies (measured here, single thread, Apple M4 Pro). A cold attempt costs TCP + TLS 1.3 setup (2 RTT) plus the request round trip plus the receiver's time; the bound is network latency and receiver speed. Worst case about 8 attempts per (event, endpoint).
• Levers:
  • connection pool per endpoint host, keep-alive — skips TCP and TLS setup after the first event (Shopify asks receivers to enable it) — steady traffic
  • per-endpoint event-type filter at the producer — no request, signature or retry for unwanted events (Standard Webhooks) — many event types
  • thin events (id and type; receiver fetches state) — fewer bytes, never stale (Stripe thin events) — large or fast-changing objects
  • spaced retries with jitter, then disable — about 8 attempts instead of a retry loop; stops work for dead endpoints after 5 days (Svix) — always
  • per-endpoint circuit breaker and concurrency cap — a slow receiver cannot pin all workers — shared worker pools
• Naive → standard → state of the art:
• POST inline in the request path, new connection, retry at once in a loop · blocks callers, retry storms against a failing receiver
• Outbox rows, worker pool, keep-alive, 8-step schedule with jitter · one warm request per event; at most about 8 attempts
• Producer-side filtering, thin events, breaker and cap per endpoint, auto-disable, events API for reconciliation · requests only for wanted events; dead endpoints cost nothing

Implementation notes

• The signed bytes are the raw body. Parsing and re-serializing JSON changes whitespace and breaks the signature; this is the most common receiver bug.
• Use per-endpoint secrets of 24-64 random bytes. Rotation: sign with old and new key, send both signatures, receivers accept any match.
• Claim rows with SELECT ... FOR UPDATE SKIP LOCKED (PostgreSQL) so workers do not block each other.
• Set explicit connect (1-5 s) and total timeouts and read only a few KB of the response; otherwise one slow receiver pins a worker.

Pitfalls

• Receiving at least once but writing non-idempotently: double charges, double emails. Dedupe by event id, or see idempotency keys.
• Assuming order. Stripe states events may arrive out of order; use object version or fetch current state.
• Tolerance of 0 or no timestamp check: no replay protection (Stripe warns against 0).
• Comparing signatures with ==: use constant time (constant-time code).
• SSRF: a customer URL pointing at 169.254.169.254 or an internal service. Check the resolved address, connect to it without a second lookup (DNS rebinding), refuse redirects; IPv6 forms that embed IPv4 (mapped, NAT64) need the same check. Best: send from a proxy such as smokescreen in an isolated subnet.
• Doing real work before answering 2xx: the sender times out and retries, so you work twice.
• Reusing the first attempt's timestamp on retries: the receiver's tolerance then rejects them.

Variants and alternatives

In production

Stripe, GitHub and Shopify send webhooks as above; Svix, Convoy and Hookdeck Outpost run sending for other platforms. Docs: Stripe (payments), GitHub (code hosting), Shopify (commerce).

Implementations

Sources

• Standard Webhooks, Specification v1.0.0. spec
• Stripe, Receive Stripe events in your webhook endpoint; Webhook signature errors. docs · signatures
• GitHub, Best practices for using webhooks; Validating webhook deliveries; Handling failed webhook deliveries. best practices · validating · failed
• Shopify, Verify webhook deliveries. docs
• Svix, Retries. docs
• H. Krawczyk, M. Bellare, R. Canetti, RFC 2104: HMAC, 1997. rfc
• OWASP, API Security Top 10 2023, API7: Server Side Request Forgery. owasp