Bot detection

Deep dive

How it works

Parts:

• Identity check — User-Agent plus verification of claimed good bots (reverse DNS with forward confirmation, or published IP ranges).
• Network signals — IP and ASN reputation, DNS blocklists, TLS fingerprint versus the declared browser.
• Rate and timing counters — per-key sliding window, inter-request gaps and their regularity.
• Behaviour and integrity signals — navigation graph (pages reached without the assets or links that lead to them), input events (behavioural biometrics), device fingerprint consistency, posture attestation.
• Scorer — rules plus a weighted log-odds sum or a fitted model, per client.
• Policy — thresholds mapping score to action; allowlist beside it.
• Feedback store — outcomes and labels from confirmed abuse, used to refit weights.

1. Resolve the client key (session, account, device id, IP prefix; use several).
2. Verified good bot: allow under a generous quota and stop.
3. Update counters: window count, last gap in a small ring.
4. Per signal, a log-odds term: rate above a reference, regularity (coefficient of variation of gaps; low is suspicious), header or TLS mismatch, behaviour and integrity flags.
5. Sum with a bias: lo = b + Σ w_i · x_i.
6. Map to allow, log, rate-limit, challenge or block; each step costs a real user more, so thresholds follow the false-positive price.
7. Record decision and signals; abuse reports, chargebacks and confirmed fraud become labels.
8. Refit weights on labels offline; run new weights in shadow mode (log only) before enforcing.

request -> [identity] -> good bot? --yes--> allow
              |no
   [network][rate/timing][behaviour][integrity]
              \      |       |       /
               +--> score (log-odds) --> thresholds
        allow | log | rate-limit | challenge | block
                          ^                    |
                          +-- labels <---------+

Good-bot check (Google's documented procedure): reverse-resolve the IP, check the name is under Google's crawler domains, forward-resolve it and require the same IP back; at volume, match the published IP-range files. Other search engines publish equivalents.

OWASP's handbook names the threats (for example OAT-005 Scalping, OAT-008 Credential Stuffing, OAT-016 Skewing); set weights per threat.

Code

use std::collections::{HashMap, HashSet, VecDeque};

#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]
enum Action { Allow, Log, RateLimit, Challenge, Block }

#[derive(Default)]
struct Client { win: VecDeque<u64>, gaps: [f64; 16], n: usize, last: u64 }

struct Scorer { clients: HashMap<u64, Client>, allow: HashSet<u64>, window_ms: u64 }

impl Scorer {
    /// One request from `id` at `now` (ms). `hdr_bad`: headers contradict the declared client.
    fn update(&mut self, id: u64, now: u64, hdr_bad: bool) -> Action {
        if self.allow.contains(&id) { return Action::Allow; } // verified crawler: rDNS + forward confirm done elsewhere
        let c = self.clients.entry(id).or_default();
        c.win.push_back(now);
        while c.win.front().map_or(false, |&t| now - t > self.window_ms) { c.win.pop_front(); }
        if c.last != 0 { c.gaps[c.n % 16] = (now - c.last) as f64; c.n += 1; }
        c.last = now;
        let rate = c.win.len() as f64 / (self.window_ms as f64 / 1000.0); // req/s
        let mut lo = -2.0 + 2.0 * (rate / 2.0).max(1.0).ln(); // (a) rate above 2 req/s
        if c.n >= 8 {                                          // (b) too-regular gaps
            let k = c.n.min(16);
            let mean = c.gaps[..k].iter().sum::<f64>() / k as f64;
            let var = c.gaps[..k].iter().map(|g| (g - mean).powi(2)).sum::<f64>() / k as f64;
            lo += 6.0 * ((0.6 - var.sqrt() / mean.max(1.0)).max(0.0) / 0.6);
        }
        if hdr_bad { lo += 2.0; }                              // (c) header-consistency flag
        match lo { x if x < 0.0 => Action::Allow, x if x < 1.0 => Action::Log,
                   x if x < 2.0 => Action::RateLimit, x if x < 4.0 => Action::Challenge, _ => Action::Block }
    }
}

struct Rng(u64);
impl Rng {
    fn f(&mut self) -> f64 { self.0 ^= self.0 << 13; self.0 ^= self.0 >> 7; self.0 ^= self.0 << 17; (self.0 >> 11) as f64 / (1u64 << 53) as f64 }
    fn normal(&mut self) -> f64 { (0..12).map(|_| self.f()).sum::<f64>() - 6.0 }
}

fn main() {
    let mut rng = Rng(0x9E3779B97F4A7C15);
    let mut s = Scorer { clients: HashMap::new(), allow: HashSet::from([9_999]), window_ms: 10_000 };
    let (mut bots_hit, mut humans_hit) = (0, 0);
    for id in 0..1000u64 {
        let (mut t, mut worst) = (1_000 + id, Action::Allow);
        for i in 0..200 { // human: bursts of 1-6 page loads, lognormal pauses, rare header quirk
            t += if i % 4 != 0 { 100 + (rng.f() * 900.0) as u64 } else { (3_000.0 * (1.2 * rng.normal()).exp()) as u64 + 500 };
            let a = s.update(id, t, rng.f() < 0.01);
            if i >= 8 && a > worst { worst = a; }
        }
        humans_hit += (worst >= Action::Challenge) as u32;
    }
    for id in 1000..2000u64 {
        let (gap, mut t, mut last) = (300 + (rng.f() * 4_000.0) as u64, 1_000 + id, Action::Allow);
        for _ in 0..200 { t += gap + (rng.f() * 0.04 * gap as f64) as u64; last = s.update(id, t, rng.f() < 0.5); }
        bots_hit += (last >= Action::Challenge) as u32;
    }
    let (mut crawler, mut t) = (Action::Block, 1_000);
    for _ in 0..200 { t += 1_000; crawler = s.update(9_999, t, true); } // regular, odd headers, but allowlisted
    println!("bots challenged/blocked {bots_hit}/1000, humans {humans_hit}/1000, crawler {crawler:?}");
    assert!(bots_hit >= 950 && humans_hit <= 20 && crawler == Action::Allow);
    s.allow.clear();
    for _ in 0..200 { t += 1_000; crawler = s.update(9_999, t, true); } // same traffic without the allowlist
    assert!(crawler >= Action::Challenge);
}

Fixed-seed run: 1,000 of 1,000 scripted clients end at challenge or block, 0 of 1,000 irregular humans reach it after warm-up, the allowlisted crawler passes, and the same traffic without the allowlist is challenged. The populations are synthetic and separable by construction: this shows the mechanics, not real accuracy.

Complexity and constants

• Amortised O(1) expiry; regularity scans at most 16 gaps. State: a timestamp deque plus 16 floats, a few hundred bytes per client.
• Measured (Apple Silicon Mac, release build, this code plus a timing loop over 1,000 clients): about 80-110 ns per update over three runs, dominated by hashing and cache misses.
• DNS verification (network round trips) is the costly step: once per IP, cache the verdict.

Efficiency

Cost: about 100 ns per update for rate, regularity and header terms (measured); DNS verification costs two lookups once per IP, then a cache hit. Bound by memory access per client, not arithmetic.

Levers:

• staged evaluation — costly signals only for grey-zone clients — when cheap terms separate most traffic.
• cache good-bot verdicts per IP — one DNS pair instead of one per request — crawlers send thousands of requests.
• fixed gap ring, incremental counters — no allocation per request — hot path.
• shard client state by key — scales across cores; see distributed rate limiting.

Naive → standard → state of the art:

• User-Agent block list · near zero cost, weak
• weighted score over rate, timing, headers with graded actions · about 100 ns per request, cheap
• layered score plus fingerprint, behaviour and integrity signals, a model fitted on labels, a challenge for the grey zone · collection and inference cost, best accuracy

Implementation notes

• Score several keys (IP prefix, session, account, device id); IP alone misreads NAT and rotation.
• Check with calibration that a score means the same abuse rate every week.
• Decay old evidence (time-decay scoring) so recovered clients are freed.
• Rate-limit verified bots too (token bucket).

Pitfalls

• Shared IPs (offices, schools, carriers, VPNs) look like one busy client; never block on IP alone.
• Accessibility: assistive technology and keyboard-only use can look machine-like; image or puzzle challenges can lock out blind, low-vision or cognitively impaired users. Offer alternatives, prefer proof-of-work or invisible checks, give an appeal route.
• Regularity also comes from honest software: monitors, polling apps, partner integrations. Allowlist by agreement.
• Labels are biased toward what you already catch; review a sample of allowed traffic.
• Signals decay as attackers adapt and browsers change; re-measure, and layer rather than rely on one.
• IPs, fingerprints and input events can be personal data: collect the minimum, truncate or hash, short retention, documented purpose.
• A wrong block is silent to you; show a reason and a way through.

Variants and alternatives

In production

Ticketing and retail sites defend drops against scalping; banks and social networks score logins and sign-ups; ad networks and analytics vendors filter invalid clicks and impressions before reporting; game operators and API providers detect scripted play and scraping. Search engines publish how to verify their crawlers.

Implementations

Sources

• OWASP, Automated Threats to Web Applications (OWASP Automated Threat Handbook). project
• Google Search Central, Verifying requests from Google crawlers and fetchers (reverse DNS, forward confirm, published IP ranges). doc
• Media Rating Council, Invalid Traffic Detection and Filtration Guidelines Addendum (GIVT and SIVT categories), update June 2020. PDF
• IAB Tech Lab, Spiders & Bots List (keeps known automated traffic out of impression counts). page