Hardware security module (HSM)

Deep dive

How it works

Parts:

• Key store — objects (key plus attributes CKA_SENSITIVE, CKA_EXTRACTABLE, CKA_SIGN, CKA_WRAP, CKA_UNWRAP) inside the boundary.
• Engine and RNG — runs mechanisms (CKM_*) on handles; generates keys internally.
• Roles and quorum — login by user or officer; M-of-N approval for admin actions.
• Interface — PKCS#11 (Cryptoki) library or a vendor/cloud API.
• Backup and sync — a key leaves a device only wrapped under a key the target also holds.
• Tamper response — zeroizes plaintext keys on intrusion (FIPS 140 Level 3).

Operations:

1. Open: C_Initialize, C_OpenSession, C_Login. All sessions of an application share the login state.
2. Create: C_GenerateKey or C_GenerateKeyPair with a template (CKA_SENSITIVE true, CKA_EXTRACTABLE false, CKA_TOKEN true to persist). Result: a CK_OBJECT_HANDLE.
3. Use: C_SignInit(handle) then C_Sign(data); likewise encrypt, decrypt, MAC. The handle is a name, not a secret; every call re-checks login and attributes.
4. Read-back refused: C_GetAttributeValue on CKA_VALUE of a sensitive or non-extractable key returns CKR_ATTRIBUTE_SENSITIVE.
5. Back up: C_WrapKey(wrapping_key, key) needs CKA_WRAP on the wrapping key and CKA_EXTRACTABLE true on the target (else CKR_KEY_UNEXTRACTABLE). Restore with C_UnwrapKey (CKA_UNWRAP).
6. Administer: M approvers for user or key changes; a tamper event wipes keys.

 app --C_Sign(handle, digest)--> [PKCS#11 lib / API] --> +--- HSM (tamper boundary) -----+
     <-------- signature ------                          | handle -> key + attributes    |
 backup <-- C_WrapKey: key encrypted under KEK --------- | engine, RNG, roles, quorum    |
                                                         +-------------------------------+

FIPS 140-3 (via ISO/IEC 19790) keeps four levels; per the FIPS 140-2 text, Level 2 adds tamper evidence, Level 3 adds response to physical access (zeroizing keys) and identity-based authentication, Level 4 a full protective envelope. Cloud services quote Level 3.

Code

A model of the interface boundary, not a secure implementation: the "mechanism" is a non-cryptographic keyed hash from std, so it shows handles, attributes, wrap and unwrap, not security. Compiled and run with the asserts below.

//! MODEL of the PKCS#11 boundary, not a secure implementation: the "crypto" is a
//! non-cryptographic keyed hash (std SipHash), so the example needs no crates.
mod hsm {
    use std::{collections::HashMap, fs::File, hash::Hasher, io::Read};

    #[derive(Clone, Copy, PartialEq, Eq, Debug)]
    pub struct Handle(u64); // CK_OBJECT_HANDLE: a name, not a secret; private field = no forging
    #[derive(Clone, Copy)]
    pub struct Attrs { pub sign: bool, pub wrap: bool, pub unwrap: bool, pub sensitive: bool, pub extractable: bool }
    #[derive(Debug, PartialEq)]
    pub enum Ck { UserNotLoggedIn, ObjectHandleInvalid, AttributeSensitive, KeyUnextractable,
                  KeyFunctionNotPermitted, WrappedKeyInvalid }

    struct Obj { key: [u8; 32], a: Attrs }
    pub struct Hsm { objs: HashMap<u64, Obj>, next: u64, pin: String, logged_in: bool }

    fn rand<const N: usize>() -> [u8; N] { // OS entropy, std only
        let mut b = [0u8; N];
        File::open("/dev/urandom").unwrap().read_exact(&mut b).unwrap();
        b
    }
    fn prf(key: &[u8], data: &[u8], ctr: u64) -> [u8; 8] { // stand-in for the CKM_* mechanism
        let mut h = std::hash::DefaultHasher::new();
        h.write(key); h.write(data); h.write_u64(ctr);
        h.finish().to_le_bytes()
    }
    fn xor_stream(key: &[u8], nonce: &[u8], data: &[u8]) -> Vec<u8> { // toy cipher: XOR is its own inverse
        data.chunks(8).enumerate().flat_map(|(i, c)| {
            let ks = prf(key, nonce, i as u64 + 1);
            c.iter().zip(ks).map(|(x, k)| x ^ k).collect::<Vec<_>>()
        }).collect()
    }

    impl Hsm {
        pub fn new(pin: &str) -> Hsm { Hsm { objs: HashMap::new(), next: 1, pin: pin.into(), logged_in: false } }
        pub fn login(&mut self, pin: &str) -> bool { self.logged_in = pin == self.pin; self.logged_in } // C_Login
        fn get(&self, h: Handle) -> Result<&Obj, Ck> {
            if !self.logged_in { return Err(Ck::UserNotLoggedIn); }
            self.objs.get(&h.0).ok_or(Ck::ObjectHandleInvalid)
        }
        fn put(&mut self, key: [u8; 32], a: Attrs) -> Handle {
            self.next += 1;
            self.objs.insert(self.next, Obj { key, a });
            Handle(self.next)
        }
        pub fn generate_key(&mut self, a: Attrs) -> Result<Handle, Ck> { // C_GenerateKey
            if !self.logged_in { return Err(Ck::UserNotLoggedIn); }
            Ok(self.put(rand(), a))
        }
        pub fn get_value(&self, h: Handle) -> Result<[u8; 32], Ck> { // C_GetAttributeValue(CKA_VALUE)
            let o = self.get(h)?;
            if o.a.sensitive || !o.a.extractable { Err(Ck::AttributeSensitive) } else { Ok(o.key) }
        }
        pub fn sign(&self, h: Handle, msg: &[u8]) -> Result<[u8; 8], Ck> { // C_SignInit + C_Sign
            let o = self.get(h)?;
            if o.a.sign { Ok(prf(&o.key, msg, 0)) } else { Err(Ck::KeyFunctionNotPermitted) }
        }
        pub fn wrap_key(&self, wrapping: Handle, target: Handle) -> Result<Vec<u8>, Ck> { // C_WrapKey
            let (w, t) = (self.get(wrapping)?, self.get(target)?);
            if !w.a.wrap { return Err(Ck::KeyFunctionNotPermitted); }
            if !t.a.extractable { return Err(Ck::KeyUnextractable); }
            let nonce: [u8; 8] = rand();
            let mut out = [&nonce[..], &xor_stream(&w.key, &nonce, &t.key)].concat();
            out.extend(prf(&w.key, &out, 0)); // integrity tag over nonce || ciphertext
            Ok(out)
        }
        pub fn unwrap_key(&mut self, unwrapping: Handle, blob: &[u8], a: Attrs) -> Result<Handle, Ck> { // C_UnwrapKey
            let w = self.get(unwrapping)?;
            if !w.a.unwrap { return Err(Ck::KeyFunctionNotPermitted); }
            if blob.len() != 48 || prf(&w.key, &blob[..40], 0) != blob[40..] { return Err(Ck::WrappedKeyInvalid); }
            let key = xor_stream(&w.key, &blob[..8], &blob[8..40]).try_into().unwrap();
            Ok(self.put(key, a)) // the template decides the new key's attributes
        }
    }
}

use hsm::*;

fn main() {
    let (mut a, vault) = (Hsm::new("1234"), Attrs { sign: true, wrap: false, unwrap: false, sensitive: true, extractable: false });
    assert_eq!(a.generate_key(vault).unwrap_err(), Ck::UserNotLoggedIn);
    assert!(!a.login("0000") && a.login("1234"));

    let signer = a.generate_key(vault).unwrap();
    assert_eq!(a.sign(signer, b"hello"), a.sign(signer, b"hello")); // deterministic stand-in
    assert_ne!(a.sign(signer, b"hello"), a.sign(signer, b"hellp"));
    assert_eq!(a.get_value(signer), Err(Ck::AttributeSensitive)); // key bytes never leave

    // Backup: a wrapping key moves an *extractable* key out as ciphertext only.
    let kek = a.generate_key(Attrs { sign: false, wrap: true, unwrap: true, ..vault }).unwrap();
    let other = a.generate_key(Attrs { sign: false, wrap: true, unwrap: true, ..vault }).unwrap();
    let svc_attrs = Attrs { extractable: true, ..vault };
    let svc = a.generate_key(svc_attrs).unwrap();
    assert_eq!(a.wrap_key(kek, signer), Err(Ck::KeyUnextractable)); // non-extractable: no backup possible
    assert_eq!(a.wrap_key(signer, svc), Err(Ck::KeyFunctionNotPermitted)); // signer lacks CKA_WRAP
    let mut blob = a.wrap_key(kek, svc).unwrap();
    assert_eq!(a.unwrap_key(other, &blob, svc_attrs), Err(Ck::WrappedKeyInvalid)); // wrong KEK

    let restored = a.unwrap_key(kek, &blob, svc_attrs).unwrap(); // new handle, same key material
    assert_ne!(restored, svc);
    assert_eq!(a.sign(restored, b"m"), a.sign(svc, b"m"));
    assert_eq!(a.get_value(restored), Err(Ck::AttributeSensitive)); // still sensitive: never in the clear
    blob[10] ^= 1; // flipped ciphertext bit
    assert_eq!(a.unwrap_key(kek, &blob, svc_attrs), Err(Ck::WrappedKeyInvalid));
}

Complexity and constants

• Per call O(1): one device operation plus one round trip. Cost drivers: number of HSM calls and bytes per call (AWS KMS Sign takes up to 4,096 bytes, a digest beyond that).
• A synchronous caller gets 1 / (round trip + device time) calls per thread; filling the device needs concurrency = throughput x latency. A networked round trip is order of milliseconds, not measured here.
• AWS CloudHSM sign rates, 2 / 3 / 6 HSMs (hsm2m.medium; multi-threaded Java client on c4.large): RSA-2048 2,000 / 3,000 / 5,000 per s; EC P-256 3,000 / 4,500 / 7,000; ML-DSA-44 80 / 140 / 190.
• Quotas: AWS KMS default 1,000 per s shared for RSA or ECC Sign (adjustable). Google Cloud HSM meters tokens, 3,000,000 per minute by default: RSA-2048 sign 1,500, P-256 sign 4,500. Keys: CloudHSM hsm2m 16,666 per cluster; Azure Managed HSM 5,000.
• Admin quorum: CloudHSM 2 to 8 approvers; Azure security domain at least 3 key pairs, quorum up to 10.

Efficiency

• Cost: bound by device throughput and round-trip latency. Measured on one Apple M4 Pro core (ed25519-dalek 2, RustCrypto p256 0.13): Ed25519 about 9 us, P-256 about 110 us per sign, so a 2-HSM cluster (3,000 P-256 signs/s) buys key safety, not speed.
• Levers:
  • Envelope encryption — HSM wraps 32-byte data keys, bulk bytes never reach it (envelope encryption) — large or many-object data.
  • Sign digests — message bytes stay off the wire (AWS 4,096 B limit) — large payloads.
  • Batch under one signature — sign a Merkle root over N items: N-fold fewer HSM calls by construction, paid with a batch window and log N proof size — receipts, timestamps, log heads.
  • Cache unwrapped data keys per window — one KMS call per window (AWS data key caching) — hot paths, trading exposure for speed.
  • Scale out — 2 to 6 HSMs gives 2,000 to 5,000 RSA-2048 signs/s (AWS), sub-linear — when the pool is the limit; one session per thread.
• Naive → standard → state of the art:
  • One synchronous HSM call per message · 1 / (round trip + device time) per thread, capped near 1,000/s by a default KMS quota.
  • Wrapped data keys with local AEAD, signed digests, pooled sessions, software verification · HSM calls scale with objects.
  • Cached keys plus Merkle-root batching · HSM calls scale with batches or windows.

Implementation notes

• Sessions are the unit of concurrency: an active operation blocks another of the same kind on that session (CKR_OPERATION_ACTIVE); use one session per thread. CloudHSM allows 1,024 PKCS#11 sessions per application.
• Set the template explicitly: CKA_SENSITIVE true, CKA_EXTRACTABLE false, only needed usage flags. Pin what a wrapping key may touch with CKA_WRAP_TEMPLATE and CKA_UNWRAP_TEMPLATE.
• Expect throttling ("HSMs are busy or throttled"): back off, size from a load test plus one spare HSM. AWS advises 2 HSMs in different AZs, 3 for durable new keys.
• Test with SoftHSM2 (brew install softhsm, softhsm2-util --init-token --slot 0 --label "t", SOFTHSM2_CONF): keys sit in files, so it tests the API, never the protection.

Pitfalls

• A key with both CKA_WRAP and CKA_DECRYPT lets an attacker wrap a sensitive key and decrypt the blob; Clulow (2003) and Bortolozzo et al. (2010) broke real tokens this way. Separate roles; use CKA_WRAP_WITH_TRUSTED and wrap templates.
• An attacker with a login can sign anything: guard the credential, log calls, restrict what is signed.
• C_UnwrapKey sets CKA_NEVER_EXTRACTABLE and CKA_ALWAYS_SENSITIVE false and defaults CKA_EXTRACTABLE to true: pass a template.
• One device is one failure: hardware loss, or lockout after losing the Azure security domain quorum.
• FIPS claims bind to a validated module, version and mode. CMVP moved FIPS 140-2 certificates to the Historical list after 2026-09-21; Google's Cloud HSM page still says 140-2 Level 3.

Variants and alternatives

In production

Payments: CloudHSM lists PCI PIN and PCI 3DS compliance. DNSSEC: SoftHSM began as the PKCS#11 store of OpenDNSSEC. Storage and databases: key-encryption keys in KMS or HSM wrap data keys. Certificate authorities and code signing: issuing keys behind PKCS#11 or a cloud HSM API. AWS KMS uses FIPS 140-3 Level 3 HSMs with no export of plaintext keys; Azure Managed HSM and Google Cloud HSM use Marvell LiquidSecurity.

Implementations

Sources

• OASIS, PKCS #11 Specification Version 3.1, 2023. spec
• NIST, FIPS 140-3, 2019. doi · FIPS 140-2. nist · CMVP
• Bortolozzo, Centenaro, Focardi, Steel, Attacking and fixing PKCS#11 security tokens, ACM CCS 2010. doi
• Clulow, On the Security of PKCS #11, CHES 2003. doi
• AWS CloudHSM: performance · quotas · HSM types · quorum
• AWS KMS: overview · request quotas · Sign. Azure Managed HSM: security domain · limits. Google Cloud: Cloud HSM · quotas
• tpm2-tools man pages: tpm2_create · attributes. SoftHSM2 README.