SSH protocol (transport, auth, channels)

Deep dive

How it works

In: a TCP stream, the server's host key, the user's credential, algorithm lists · Out: authenticated, encrypted channels (shells, commands, subsystems, forwards) · State: per direction a sequence number and keys, the session id, per channel two window sizes · Cost: one key exchange and two signatures per connection, then one cipher pass plus MAC per packet

1. Version exchange. Each side sends SSH-2.0-softwareversion SP comments CR LF (at most 255 characters); both strings enter the exchange hash (RFC 4253 section 4).
2. Algorithm negotiation. Both send SSH_MSG_KEXINIT: a 16-byte cookie and name-lists for key exchange, host key, cipher, MAC and compression. Per category the first algorithm on the client's list that the server supports wins (section 7.1).
3. Key exchange. For curve25519-sha256 (RFC 8731) the client sends an ephemeral X25519 key; the server replies with its own, its host key and a signature over the exchange hash H. H covers both identification strings, both KEXINIT payloads, the host key, both ephemeral keys and the shared secret K (ECDH). The client checks the signature against a trusted host key (ssh-ed25519, RFC 8709, Ed25519).
4. Keys. The first H is the session id. Each key is a hash of K || H || letter || session_id: A and B the IVs, C and D the encryption keys, E and F the integrity keys (client to server, then server to client), extended by re-hashing if longer (section 7.2). This is SSH's own hash construction, not HKDF.
5. NEWKEYS. SSH_MSG_NEWKEYS from each side switches that direction to the new keys.
6. User authentication. The client requests service ssh-userauth and tries publickey (a signature over data that includes the session id), password, hostbased or keyboard-interactive (RFC 4256); the server lists methods that may continue.
7. Channels. SSH_MSG_CHANNEL_OPEN carries a type, an initial window size and a maximum packet size; the peer confirms with its own. A sender may send only as many data bytes as the window allows; SSH_MSG_CHANNEL_WINDOW_ADJUST adds credit, up to 2^32 - 1 (RFC 4254 section 5.2). This is credit-based flow control per channel.
8. Requests. On a session channel the client asks for a pty, exec, shell or subsystem (SFTP is the subsystem sftp). Port forwarding uses direct-tcpip (local forward) and tcpip-forward with forwarded-tcpip (remote forward); dynamic forwarding is a local SOCKS listener opening one direct-tcpip channel per connection.

Binary packet. After NEWKEYS each message is packet_length (u32) | padding_length (u8) | payload | random padding | MAC. Padding makes the length a multiple of max(cipher block, 8) with at least 4 padding bytes (section 6); implementations must accept packets up to 35,000 bytes (section 6.1). The MAC covers sequence_number || unencrypted packet; the sequence number is an implicit 32-bit counter, per packet, wrapping at 2^32 (section 6.4). AEAD modes (chacha20-poly1305@openssh.com, AES-GCM) replace cipher and MAC with one operation.

Rekeying. Either side can start a new key exchange under the current keys; the session id stays. RFC 4253 recommends it after each gigabyte or hour (section 9).

Host key trust. The protocol does not say how a client learns the host key. OpenSSH defaults to trust on first use (StrictHostKeyChecking ask): show the fingerprint, pin it in known_hosts, refuse on change. Alternatives: a host certificate signed by a CA the client trusts, or SSHFP records in DNSSEC-signed DNS (RFC 4255, VerifyHostKeyDNS).

User certificates. An OpenSSH CA signs a user key with principals and a validity window; sshd trusts the CA via TrustedUserCAKeys, so no per-user key lists (draft-miller-ssh-cert).

Terrapin and strict KEX. Sequence numbers continued across the key switch, so an on-path attacker could insert messages before encryption and delete as many of the first encrypted ones without breaking a MAC (Bäumer, Brinkmann, Schwenk, USENIX Security 2024). OpenSSH 9.6 added "strict KEX", used when both sides support it: an unexpected message during key exchange ends the connection, and sequence numbers reset to zero at NEWKEYS.

Post-quantum. Hybrid exchanges add a lattice KEM to X25519 so recorded traffic resists a future quantum computer (post-quantum KEM). OpenSSH 8.9 added sntrup761x25519-sha512@openssh.com, 9.0 made it default, 9.9 added mlkem768x25519-sha256, 10.0 made that default; 10.1 warns on non-PQ exchanges (release notes).

Code

Rust std only: RFC 4253 section 6 framing, a stub MAC (toy hash, not secure; real code uses HMAC or an AEAD) and per-direction sequence numbers with the strict-KEX reset as a flag. Cipher "none", block 8 or 16.

// RFC 4253 section 6 binary packet protocol, cipher = none (block 8), MAC pluggable.
#[derive(Debug, PartialEq)]
enum PErr { Short, Oversize, Misaligned, ShortPadding, BadPadLen, BadMac }
const MAX_PACKET: usize = 35000; // section 6.1: whole packet incl. mac
/// Toy stand-in for HMAC: mixes sequence number and packet bytes. NOT secure.
fn stub_mac(seq: u32, pkt: &[u8]) -> Vec<u8> {
    let mut h = 0x811c9dc5u32 ^ seq;
    for &b in pkt { h = (h ^ b as u32).wrapping_mul(0x0100_0193); }
    h.to_be_bytes().to_vec()
}
struct Dir { seq: u32, block: usize, strict: bool, mac: fn(u32, &[u8]) -> Vec<u8>, mac_len: usize, rng: u64 }
impl Dir {
    fn new(block: usize, strict: bool) -> Dir { // block = max(cipher block, 8)
        Dir { seq: 0, block: block.max(8), strict, mac: stub_mac, mac_len: 4, rng: 0x9e3779b97f4a7c15 }
    }
    fn rand(&mut self) -> u8 { // xorshift64; real code: OS CSPRNG
        self.rng ^= self.rng << 13; self.rng ^= self.rng >> 7; self.rng ^= self.rng << 17; (self.rng >> 32) as u8
    }
    fn encode(&mut self, payload: &[u8]) -> Vec<u8> {
        let mut pad = self.block - (5 + payload.len()) % self.block; // 1..=block
        if pad < 4 { pad += self.block; } // at least 4 padding bytes
        let len = 1 + payload.len() + pad; // packet_length excludes itself and the mac
        let mut p = Vec::with_capacity(4 + len + self.mac_len);
        p.extend_from_slice(&(len as u32).to_be_bytes());
        p.push(pad as u8);
        p.extend_from_slice(payload);
        for _ in 0..pad { p.push(self.rand()); }
        let tag = (self.mac)(self.seq, &p); // MAC over seq || unencrypted packet
        p.extend_from_slice(&tag);
        self.seq = self.seq.wrapping_add(1); // counts every packet, wraps at 2^32
        p
    }
    /// Returns (payload, bytes consumed). Sequence number moves only on success.
    fn decode(&mut self, buf: &[u8]) -> Result<(Vec<u8>, usize), PErr> {
        let n = buf.get(..4).ok_or(PErr::Short)?;
        let len = u32::from_be_bytes([n[0], n[1], n[2], n[3]]) as usize;
        let total = 4usize.saturating_add(len).saturating_add(self.mac_len);
        if total > MAX_PACKET { return Err(PErr::Oversize); } // check before allocating or waiting
        if (4 + len) % self.block != 0 || len < 12 { return Err(PErr::Misaligned); }
        if buf.len() < total { return Err(PErr::Short); }
        let pad = buf[4] as usize;
        if pad < 4 { return Err(PErr::ShortPadding); }
        if pad + 1 > len { return Err(PErr::BadPadLen); }
        let (pkt, tag) = buf[..total].split_at(4 + len);
        let want = (self.mac)(self.seq, pkt);
        if want.iter().zip(tag).fold(0u8, |a, (x, y)| a | (x ^ y)) != 0 { return Err(PErr::BadMac); }
        self.seq = self.seq.wrapping_add(1);
        Ok((pkt[5..4 + len - pad].to_vec(), total))
    }
    fn newkeys(&mut self) { if self.strict { self.seq = 0; } } // after NEWKEYS; strict KEX only
}
fn put_len(w: &mut [u8], len: usize) { w[..4].copy_from_slice(&(len as u32).to_be_bytes()); }
fn main() {
    for block in [8, 16] {
        for n in 0..70 { // alignment and round trip for many sizes
            let (mut t, mut r) = (Dir::new(block, false), Dir::new(block, false));
            let msg: Vec<u8> = (0..n).map(|i| i as u8).collect();
            let wire = t.encode(&msg);
            assert_eq!((wire.len() - t.mac_len) % block, 0);
            assert!(wire[4] >= 4); // at least 4 padding bytes
            assert_eq!(r.decode(&wire), Ok((msg, wire.len())));
        }
    }
    let (mut t, mut r) = (Dir::new(8, false), Dir::new(8, false));
    let mut w = t.encode(b"abc"); w[4] = 3;
    assert_eq!(r.decode(&w), Err(PErr::ShortPadding));
    let mut w = t.encode(b"abc"); put_len(&mut w, 40000);
    assert_eq!(r.decode(&w), Err(PErr::Oversize));
    let mut w = t.encode(b"abc"); put_len(&mut w, 13);
    assert_eq!(r.decode(&w), Err(PErr::Misaligned));
    let mut w = t.encode(b"abc"); let l = w.len(); w[l - 1] ^= 1;
    assert_eq!(r.decode(&w), Err(PErr::BadMac));
    // sequence numbers wrap at 2^32; only strict KEX resets them at NEWKEYS
    let (mut t, mut r) = (Dir::new(8, false), Dir::new(8, false));
    (t.seq, r.seq) = (u32::MAX, u32::MAX);
    let w = t.encode(b"x"); r.decode(&w).unwrap();
    assert_eq!((t.seq, r.seq), (0, 0));
    t.seq = 5; t.newkeys(); assert_eq!(t.seq, 5);
    let mut s = Dir::new(8, true); s.seq = 5; s.newkeys(); assert_eq!(s.seq, 0);
}

Compiled with stable rustc, asserts pass. Timing (1 KiB payload, 2 million iterations, Apple M4 Pro, two runs): about 110 ns per encode+decode with a constant-tag MAC stub, about 2.2 µs with the byte-serial toy hash above: the MAC dominates, not the framing.

Complexity and constants

• Setup: several round trips (ECDH, then auth) before data flows; not measured here.
• Per packet: 5 header bytes, 4-255 padding bytes, plus a MAC or tag.
• Cost drivers: signatures and DH at connect (one connection per command is the common waste), one syscall per packet if writes are not coalesced, stalls when the channel window is below bandwidth times RTT.

Efficiency

• Cost: at connect two signatures, one DH per side, several round trips; then one cipher pass plus MAC per byte. Measured here: framing about 110 ns per 1 KiB packet. Bound: setup latency, then cipher throughput.
• Levers:
  • connection multiplexing (ControlMaster, ControlPersist) — later sessions skip key exchange and auth — many short commands to one host
  • AEAD ciphers — one pass instead of cipher plus MAC — bulk transfer
  • ProxyJump — credential stays local — bastion access
• Naive → standard → state of the art:
  • new connection per command · full key exchange and auth each time
  • one long session with several channels · one key exchange, per-channel windows
  • multiplexed master connection · later invocations cost a local socket round trip

Implementation notes

• Validate packet_length before allocating or waiting: it is attacker-chosen.
• Compare MACs in constant time (constant-time code); coalesce small writes (a keystroke is one packet).
• Keep ForwardAgent no (default); use ProxyJump.

Pitfalls

• Trust on first use: users accept unseen fingerprints; a changed key needs investigating, not StrictHostKeyChecking no.
• A forwarded agent on a compromised host lets its root user sign as you.
• Terrapin: the paper broke integrity for three widely used encryption modes and found 71.6% of scanned servers offering one; both ends need strict KEX (OpenSSH 9.6+).
• AllowTcpForwarding and PasswordAuthentication default to yes in sshd: shell users can tunnel into the server's network, and passwords invite guessing.

Variants and alternatives

In production

• OpenSSH is the standard remote shell on Linux, BSD and macOS; GitHub documents SSH keys for access.
• Routers, switches and appliances expose SSH; CI systems deploy over it with deploy keys.
• Libraries embed the protocol: Rust russh, Python paramiko, Go x/crypto/ssh, C libssh.

Implementations

Sources

• Ylonen, Lonvick, SSH Architecture, Authentication, Transport, Connection, RFC 4251-4254, 2006. 4251 · 4252 · 4253 · 4254
• Schlyter, Griffin, Using DNS to Securely Publish SSH Key Fingerprints, RFC 4255. RFC · Cusack, Forssen, RFC 4256. RFC
• Adamantiadis, Josefsson, Baushke, SSH Key Exchange Method Using Curve25519 and Curve448, RFC 8731, 2020. RFC · Harris, Velvindron, Ed25519 and Ed448 Public Key Algorithms for SSH, RFC 8709. RFC
• Bäumer, Brinkmann, Schwenk, Terrapin Attack: Breaking SSH Channel Integrity By Sequence Number Manipulation, USENIX Security 2024. paper · USENIX
• OpenSSH, Release notes. openssh.com · PROTOCOL. GitHub
• IETF drafts: strict KEX · certificates · chacha20-poly1305
• OpenBSD, ssh_config(5) and sshd_config(5) (defaults). ssh_config · sshd_config
• GitHub Docs, About SSH. docs · SSH over HTTPS port