Reverse proxy

Deep dive

How it works

Parts:

• listener + TLS termination — accepts sockets, handshakes, hands plaintext on
• request parser + router — strict HTTP/1.1 parsing; picks a pool by host and path
• upstream pool — addresses, health state (probes, passive ejection), policy (round robin, two choices, hash)
• connection pool — keep-alive connections to upstreams
• forwarder — strips hop-by-hop headers, adds Via and client address, sets Host
• response streamer — copies chunks as they arrive; client backpressure slows the upstream read
• policy hooks — timeouts (connect, read, idle), retries, cache lookup, rate limit

Per request:

1. Read the size-limited head; reject ambiguity (duplicate Content-Length, unhandled Transfer-Encoding, malformed fields).
2. Route to a pool; apply rate limit and cache lookup (a hit ends here).
3. Pick a healthy upstream; take a pooled connection or connect with a timeout. A failed connect marks it down and tries the next (nothing was sent, so the retry is safe).
4. Rewrite the head (drop hop-by-hop, set forwarding fields), send it, stream the request body.
5. Await the first response bytes under a read timeout: silence is 504, an empty or invalid answer 502.
6. Stream the response; return the upstream connection to the pool if the message ended cleanly.

 clients ──TLS──► [listener] ─► [parse/route] ─► [pick upstream] ─► [conn pool] ──► upstream A
                      │              │ cache/rate hooks   ▲ health        │ ─────► upstream B
                      │              ▼                    └─ probes ◄─────┘
                      └◄────────── streamed response (backpressure) ◄──────────────┘

Code

use std::io::{self, BufRead, BufReader, Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering::Relaxed};
use std::sync::Arc;
use std::time::Duration;

// always dropped (RFC 9110 §7.6.1 list, plus the older RFC 2616 hop-by-hop set)
const HOP: [&str; 9] = ["connection", "proxy-connection", "keep-alive", "proxy-authenticate",
    "proxy-authorization", "te", "trailer", "transfer-encoding", "upgrade"];

struct Proxy { ups: Vec<SocketAddr>, down: Vec<AtomicBool>, rr: AtomicUsize }

fn status(c: &mut TcpStream, s: &str) -> io::Result<()> {
    write!(c, "HTTP/1.1 {s}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
}

fn handle(p: &Proxy, mut c: TcpStream) -> io::Result<()> {
    let peer = c.peer_addr()?.ip();
    let mut r = BufReader::new(c.try_clone()?);
    let mut start = String::new();
    r.read_line(&mut start)?;
    let (mut hdrs, mut size) = (Vec::<(String, String)>::new(), start.len());
    loop {
        let mut l = String::new();
        size += r.read_line(&mut l)?;
        if l.is_empty() || size > 16 * 1024 { return status(&mut c, "431 Request Header Fields Too Large"); }
        let l = l.trim_end();
        if l.is_empty() { break; }
        match l.split_once(':') { // no space before the colon, no obs-fold: the strictness that stops smuggling
            Some((k, v)) if !k.is_empty() && !k.contains(char::is_whitespace) =>
                hdrs.push((k.to_ascii_lowercase(), v.trim().to_string())),
            _ => return status(&mut c, "400 Bad Request"),
        }
    }
    let mut cl = hdrs.iter().filter(|h| h.0 == "content-length").map(|h| h.1.parse::<u64>());
    let body_len = match (cl.next(), cl.next()) {
        (None, _) => 0,
        (Some(Ok(n)), None) => n,
        _ => return status(&mut c, "400 Bad Request"), // bad or duplicate Content-Length
    };
    if hdrs.iter().any(|h| h.0 == "transfer-encoding") { return status(&mut c, "501 Not Implemented"); }
    // hop-by-hop = the fixed list plus every token the sender named in Connection
    let named: Vec<String> = hdrs.iter().filter(|h| h.0 == "connection")
        .flat_map(|h| h.1.split(',').map(|t| t.trim().to_ascii_lowercase()).collect::<Vec<_>>()).collect();
    let mut head = start.trim_end().to_string() + "\r\n";
    for (k, v) in &hdrs {
        // we are the edge: client-supplied forwarding headers are replaced, not trusted
        if HOP.contains(&k.as_str()) || named.contains(k) || k == "x-forwarded-for" || k == "via" { continue; }
        head += &format!("{k}: {v}\r\n");
    }
    head += &format!("x-forwarded-for: {peer}\r\nvia: 1.1 kernlore\r\nconnection: close\r\n\r\n");
    let (n, first) = (p.ups.len(), p.rr.fetch_add(1, Relaxed));
    let mut up = None;
    for i in 0..n { // round robin, skipping upstreams marked down; connect failures are safe to retry
        let k = (first + i) % n;
        if p.down[k].load(Relaxed) { continue; }
        match TcpStream::connect_timeout(&p.ups[k], Duration::from_millis(300)) {
            Ok(s) => { up = Some(s); break; }
            Err(_) => p.down[k].store(true, Relaxed),
        }
    }
    let Some(mut up) = up else { return status(&mut c, "502 Bad Gateway") };
    up.set_read_timeout(Some(Duration::from_secs(5)))?;
    up.write_all(head.as_bytes())?;
    io::copy(&mut r.by_ref().take(body_len), &mut up)?; // Content-Length body, streamed
    let mut buf = [0u8; 16 * 1024];
    match up.read(&mut buf) { // first read decides 504 (silent) vs 502 (closed) while we can still change the answer
        Ok(0) => return status(&mut c, "502 Bad Gateway"),
        Ok(k) => c.write_all(&buf[..k])?,
        Err(_) => return status(&mut c, "504 Gateway Timeout"),
    }
    io::copy(&mut up, &mut c)?; // stream the rest, no full buffering; the client socket applies backpressure
    Ok(())
}

fn serve(l: TcpListener, ups: Vec<SocketAddr>) {
    let down = ups.iter().map(|_| AtomicBool::new(false)).collect();
    let p = Arc::new(Proxy { ups, down, rr: AtomicUsize::new(0) });
    for c in l.incoming().flatten() {
        let p = p.clone();
        std::thread::spawn(move || { let _ = handle(&p, c); });
    }
}

fn main() {
    // upstream: answers with its name and the request head it received; stops after `n` connections
    let upstream = |name: &'static str, n: usize| {
        let l = TcpListener::bind("127.0.0.1:0").unwrap();
        let addr = l.local_addr().unwrap();
        (addr, std::thread::spawn(move || for mut c in l.incoming().take(n).flatten() {
            let (mut got, mut line) = (String::new(), String::new());
            let mut r = BufReader::new(c.try_clone().unwrap());
            while r.read_line(&mut line).unwrap() > 2 { got += &line; line.clear(); }
            let body = format!("{name}\n{got}");
            write!(c, "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()).unwrap();
        }))
    };
    let (a, _ha) = upstream("A", 100);
    let (b, hb) = upstream("B", 2);
    let l = TcpListener::bind("127.0.0.1:0").unwrap();
    let front = l.local_addr().unwrap();
    std::thread::spawn(move || serve(l, vec![a, b]));
    let get = || {
        let mut c = TcpStream::connect(front).unwrap();
        c.write_all(b"GET /x HTTP/1.1\r\nHost: h\r\nConnection: keep-alive, X-Secret\r\nKeep-Alive: 5\r\nX-Secret: 1\r\nX-Forwarded-For: 6.6.6.6\r\n\r\n").unwrap();
        let mut s = String::new();
        c.read_to_string(&mut s).unwrap();
        s
    };
    let who: Vec<String> = (0..4).map(|_| get()).map(|r| r.split("\r\n\r\n").nth(1).unwrap().to_string()).collect();
    let names: Vec<&str> = who.iter().map(|w| &w[..1]).collect();
    assert_eq!(names, ["A", "B", "A", "B"]); // round robin
    let w = &who[0];
    assert!(w.contains("x-forwarded-for: 127.0.0.1") && !w.contains("6.6.6.6"));
    assert!(w.contains("via: 1.1 kernlore") && w.contains("host: h"));
    assert!(!w.contains("x-secret") && !w.contains("keep-alive"));
    hb.join().unwrap(); // B's listener is now closed
    for _ in 0..4 { assert!(get().contains("200 OK")); } // failover: every answer still comes from A
}

Content-Length bodies only, Connection: close on both sides (no pool), no TLS, no probe: a down upstream stays down. The test in main asserts round robin, header stripping and failover; it passes on Rust 1.99.

Complexity and constants

Parse O(head bytes), pick O(1), copy O(body bytes). Cost drivers: a new TCP (and TLS) connection per upstream request, a read/write syscall pair per chunk, a thread or task per connection, buffered bodies (memory = size × concurrent requests). Code above: 16 KiB read buffer per connection.

Efficiency

• Cost: measured by me on Apple M4 Pro, loopback, sequential, new connection per request, 5,000 requests after 1,000 warm-up, this code vs a trivial 2-byte upstream: median 80 µs direct, 170 µs via proxy, so about 90 µs added (second connect, thread spawn, parse, rewrite). Bound: syscalls and handshakes. On a real network the upstream round trip dominates.
• Levers:
  • share one upstream pool across workers — one customer's reuse ratio went from 87.1 % to 99.92 %, new origin connections down 160× (Cloudflare Pingora blog) — upstream hops over a network or TLS
  • stream instead of buffer — nginx proxy_buffering off passes data as received; default buffers to memory or disk (nginx docs) — large downloads, slow clients
  • bound retries — nginx proxy_next_upstream_tries; HAProxy retries only connect failures by default (docs) — always
  • event loop instead of thread per connection — Pingora about 70 % less CPU and 67 % less memory than the old service at equal traffic, partly architecture (Cloudflare blog) — many connections
• Naive → standard → state of the art:
  • thread per connection, connect per request, buffered bodies · about 90 µs added (measured above), memory grows with body size
  • event-loop workers, keep-alive pools, streamed bodies, passive ejection · no handshake per request; cost is parse, rewrite, one copy per chunk
  • cross-thread pools, HTTP/2 upstream, zero-copy relay of opaque bodies · Pingora serves over 40 million requests/s at Cloudflare (README)

Implementation notes

• Hop-by-hop removal is a union: the fixed list (Connection, Proxy-Connection, Keep-Alive, TE, Transfer-Encoding, Upgrade, ...) plus every token in any Connection header.
• Host: pass the client's (apps build URLs from it) or rewrite to the upstream name (nginx defaults to $proxy_host); decide explicitly.
• Forwarding: Forwarded (RFC 7239, for=192.0.2.43, IPv6 quoted) or X-Forwarded-For. At the edge replace client values; behind trusted proxies append and count only your own hops from the right (Envoy: xff_num_trusted_hops).
• Three timeouts: connect, between reads (nginx: between two successive reads, not the whole response), and pooled-idle, shorter than the upstream's own.
• Never reuse a connection after a partial body or early upstream close.
• Retry: a proxy MUST NOT automatically retry non-idempotent requests (RFC 9110); cap tries, add jitter, add a circuit breaker.

Pitfalls

• Request smuggling: proxy and upstream disagree where a message ends (Content-Length vs Transfer-Encoding, whitespace in names). Reject ambiguous heads, re-serialise, same rules at every hop (RFC 9112 §11.2).
• Spoofed X-Forwarded-For: trusting the leftmost value lets clients pick their IP for limits and allowlists.
• Open proxy: routing by request host or absolute URI to any target. Route only to configured pools.
• Buffered large bodies: memory blow-up and slow-client holds. Cap and stream.
• Retry storms: 3 layers each making 3 attempts is 27 attempts on an overloaded upstream. One retry layer, retry budgets.
• 502 vs 504: 502 = invalid or no answer, 504 = no timely answer (RFC 9110 §15.6). Only before the first byte can the status still change.
• Dropped Upgrade: breaks WebSockets unless forwarded deliberately.

Variants and alternatives

In production

• nginx, HAProxy, Caddy and Traefik at edge and API tiers; Envoy at the edge and as per-service sidecar; Pingora in front of Cloudflare's origins. Download edges add caching; model-serving routers pick a replica by load.

Implementations

Sources

• R. Fielding, M. Nottingham, J. Reschke, HTTP Semantics, RFC 9110, 2022, §7.6 (Connection, Via), §9.2.2, §15.6. rfc
• HTTP/1.1, RFC 9112, 2022, §6.3, §11.2. rfc
• A. Petersson, M. Nilsson, Forwarded HTTP Extension, RFC 7239, 2014. rfc
• nginx docs, ngx_http_proxy_module and ngx_http_upstream_module
• HAProxy configuration manual: option forwardfor, retry-on
• Envoy docs: connection pooling, HTTP headers (XFF)
• Cloudflare, How we built Pingora, 2022 · code