Data masking

Deep dive

How it works

Parts:

• Rule table — one rule per column or id domain (cust, phone), versioned.
• Masking key — secret HMAC key per environment, in a secrets manager, never shipped with the copy.
• Static masker — batch/ETL/CDC job inside the production zone; only masked rows leave.
• Dynamic masker — database policy, view or proxy returning original or mask by role.
• Token vault (optional) — random token to original mapping behind its own access control.
• Log redactor — masks where the log record is built.

1. Classify columns: direct identifiers (email, card), quasi-identifiers (ZIP, birth date), sensitive attributes. Give each id space one domain name.
2. Pick a technique: nulling, redaction (fixed text), partial mask (last 4), substitution (fake value from a list), shuffling (permute a column across rows: keeps the distribution, breaks row links), HMAC pseudonym, token (vault lookup), format-preserving encryption (reversible, see FPE), date shift (random per-person offset, keeps intervals).
3. Static: read, apply rules, write the copy; check that no column lacks a rule before release.
4. Dynamic: attach the rule to the column; the engine returns original or mask per role.
5. Referential consistency: same domain, same key, in every table and every refresh.
6. Rotate the key only with a full re-mask; a new key breaks joins to older copies.

prod --[rules+key]--> static masker --> masked copy (dev/test/analytics)
prod --[role?]------> dynamic policy: analyst -> mask, dpo -> original
log record ---------> redactor ------> log store

SQL Server dynamic masking offers default(), email(), random(), partial(prefix, padding, suffix) and datetime() (2022+); data is unchanged, UNMASK shows originals. Snowflake masking policies are expressions on CURRENT_ROLE() applied wherever the column appears, join and WHERE predicates included, so a constant mask makes all masked keys equal. PostgreSQL Anonymizer does both modes.

Code

Rust, per-column rules with HMAC-SHA256 pseudonyms via the direct hmac 0.13 / sha2 0.11 API. Compiled and run; the asserts check join consistency, kept formats, last-4 mask, interval-preserving date shift and key separation.

use hmac::{Hmac, KeyInit, Mac};
use sha2::Sha256;

type H = Hmac<Sha256>;

pub enum Rule {
    Keep,
    Null,
    Redact,                                   // fixed token, hides even the length
    KeepLast(usize),                          // "**** **** **** 1111"
    PseudoDigits(&'static str),               // same digit/separator layout, one-way
    PseudoId(&'static str),                   // 64-bit hex id, "u_9f1c..."
    DateShift { subject: usize, max_days: i64 }, // per-subject offset, ISO dates
}

pub struct Masker { keyed: H } // HMAC state after the key blocks; cloned per value

impl Masker {
    pub fn new(key: &[u8]) -> Self { Masker { keyed: H::new_from_slice(key).unwrap() } }

    /// HMAC-SHA256(key, domain || 0x00 || value || counter); the domain separates id spaces.
    fn prf(&self, domain: &str, value: &str, ctr: u8) -> [u8; 32] {
        let mut m = self.keyed.clone(); // skips re-hashing the two key pads
        m.update(domain.as_bytes());
        m.update(&[0]);
        m.update(value.as_bytes());
        m.update(&[ctr]);
        m.finalize().into_bytes().into()
    }

    pub fn apply(&self, rules: &[Rule], row: &[&str]) -> Vec<Option<String>> {
        rules.iter().zip(row).map(|(rule, &v)| match rule {
            Rule::Keep => Some(v.to_string()),
            Rule::Null => None,
            Rule::Redact => Some("[redacted]".into()),
            Rule::KeepLast(n) => {
                let total = v.chars().filter(|c| c.is_alphanumeric()).count();
                let mut seen = 0;
                Some(v.chars().map(|c| {
                    if !c.is_alphanumeric() { return c; }
                    seen += 1;
                    if seen > total.saturating_sub(*n) { c } else { '*' }
                }).collect())
            }
            Rule::PseudoDigits(d) => {
                let (mut ctr, mut i, mut block) = (0u8, 32, [0u8; 32]);
                Some(v.chars().map(|c| {
                    if !c.is_ascii_digit() { return c; }
                    loop { // rejection sampling: bytes >= 250 would bias %10
                        if i == 32 { block = self.prf(d, v, ctr); ctr += 1; i = 0; }
                        i += 1;
                        if block[i - 1] < 250 { return (b'0' + block[i - 1] % 10) as char; }
                    }
                }).collect())
            }
            Rule::PseudoId(d) => {
                let b = self.prf(d, v, 0);
                let mut s = String::from("u_");
                for x in &b[..8] { for n in [x >> 4, x & 15] { s.push(char::from_digit(n as u32, 16).unwrap()); } }
                Some(s)
            }
            Rule::DateShift { subject, max_days } => {
                let b = self.prf("dateshift", row[*subject], 0);
                let r = u64::from_be_bytes(b[..8].try_into().unwrap()) % (2 * *max_days as u64);
                let off = r as i64 - max_days + (r as i64 >= *max_days) as i64; // never 0
                let (y, m, d) = civil(iso_days(v) + off);
                Some(format!("{y:04}-{m:02}-{d:02}"))
            }
        }).collect()
    }
}

// Howard Hinnant's civil-date algorithms (proleptic Gregorian).
fn iso_days(s: &str) -> i64 {
    let p: Vec<i64> = s.split('-').map(|x| x.parse().unwrap()).collect();
    let (y, m, d) = (p[0], p[1], p[2]);
    let y = y - (m <= 2) as i64;
    let (era, yoe) = (y.div_euclid(400), y.rem_euclid(400));
    let doy = (153 * (m + if m > 2 { -3 } else { 9 }) + 2) / 5 + d - 1;
    era * 146097 + yoe * 365 + yoe / 4 - yoe / 100 + doy - 719468
}
fn civil(z: i64) -> (i64, i64, i64) {
    let z = z + 719468;
    let (era, doe) = (z.div_euclid(146097), z.rem_euclid(146097));
    let yoe = (doe - doe / 1460 + doe / 36524 - doe / 146096) / 365;
    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
    let mp = (5 * doy + 2) / 153;
    let d = doy - (153 * mp + 2) / 5 + 1;
    let m = if mp < 10 { mp + 3 } else { mp - 9 };
    (yoe + era * 400 + (m <= 2) as i64, m, d)
}

fn main() {
    let mk = Masker::new(&[7; 32]);
    // customers(id, phone, card, dob, note) and orders(customer_id, placed)
    let cust = [Rule::PseudoId("cust"), Rule::PseudoDigits("phone"), Rule::KeepLast(4),
                Rule::DateShift { subject: 0, max_days: 30 }, Rule::Null];
    let ord = [Rule::PseudoId("cust"), Rule::DateShift { subject: 0, max_days: 30 }];
    let c = mk.apply(&cust, &["C-1001", "+49 170 1234567", "4111 1111 1111 1111", "1990-03-01", "VIP"]);
    let o1 = mk.apply(&ord, &["C-1001", "2026-03-10"]);
    let o2 = mk.apply(&ord, &["C-1001", "2026-04-09"]);
    let other = mk.apply(&ord, &["C-1002", "2026-03-10"]);

    assert_eq!(c[0], o1[0]); // join consistency: same value + domain -> same pseudonym
    assert_ne!(c[0], other[0]); // different customers stay distinct
    assert_eq!(c[0].as_deref().map(str::len), Some(18)); // "u_" + 16 hex
    let ph = c[1].clone().unwrap(); // format kept: digits stay digits, separators stay
    assert_ne!(ph, "+49 170 1234567");
    assert!(ph.starts_with("+") && ph.as_bytes()[3] == b' ' && ph.len() == 15);
    assert_eq!(c[2].as_deref(), Some("**** **** **** 1111"));
    assert_eq!(c[4], None);
    // dates: one offset per subject, so intervals inside a subject survive
    let (a, b) = (iso_days(o1[1].as_ref().unwrap()), iso_days(o2[1].as_ref().unwrap()));
    assert_eq!(b - a, 30);
    assert!((a - iso_days("2026-03-10")).abs() <= 30 && a != iso_days("2026-03-10"));
    // a different key gives unrelated pseudonyms
    let mk2 = Masker::new(&[8; 32]);
    assert_ne!(mk2.apply(&ord, &["C-1001", "2026-03-10"])[0], o1[0]);
}

Complexity and constants

• Per value O(1): one HMAC (short input: 2 SHA-256 compressions with the key pads cached) plus formatting; static runs stream rows in O(1) memory.
• A 64-bit pseudonym collides around 2^32 (4.3e9) distinct values; at 1e8 ids the chance is about n²/2^65, order of 3e-4. Use 128 bits for ids that must stay unique.
• Same-length digit pseudonyms are not injective: 1e8 ten-digit phones into 1e10 slots give order of 5e5 colliding pairs. Use FPE when you need a bijection.
• A vault adds a stored mapping and a network lookup per value.

Efficiency

• Cost: measured here, Apple M4 Pro, release, one thread, 200k short values: pseudonym id order of 180 ns, digit pseudonym 155 ns, last-4 mask 70 ns, date shift 220 ns; about 5M pseudonyms/s per core. Bound: compute (static), per-row calls (dynamic), round trips (vault).
• Levers:
  • Clone cached keyed HMAC state per value, no per-byte format! — pseudonym 474 ns → 179 ns (measured here, both changes) — any bulk run.
  • Mask inside the engine — no extra bytes leave the database (Snowflake, SQL Server) — dynamic masking.
  • Shard static jobs by key range — HMAC is stateless, so linear in cores — large copies.
  • Memoise per-subject offsets and repeated values — one HMAC per distinct subject — wide tables with repeated ids.
  • Batch vault calls — one round trip per batch — tokenization.
• Naive → standard → state of the art:
  • Regex-scrub a dump per team · whole-file passes, missed columns, no join consistency.
  • Rule table + keyed HMAC in a streaming job · order of 5M values/s per core, joins preserved.
  • Role-aware policy in the engine over a single stored copy · no second copy, a function call per row; safety rests on access control.

Implementation notes

• Hash domain || 0x00 || value; without domain separation, equal values in different id spaces link.
• Normalise first (trim, lowercase emails) or A@x.com and a@x.com get different pseudonyms.
• Run the static masker inside the production zone; copy only its output.
• One-way functions for copies; tokens or FPE only where someone must reverse.
• Logs: redact where the record is built (typed fields, allow-list), not by regex afterwards; OWASP says such data should be removed, masked, sanitized, hashed or encrypted.
• Date shift: one offset per subject, never per row, never zero; weekday and season patterns change.

Pitfalls

• Masking is not anonymization. ZIP + gender + birth date identified most Americans (Sweeney 2000; 87 % from memory); a generative model estimates 99.98 % re-identifiable from 15 demographic attributes (Rocher et al. 2019). Generalise quasi-identifiers or check k-anonymity.
• Unsalted hash of a small domain: enumerating 1e9 SSNs takes minutes, order of.
• Dynamic masking leaks through predicates: Microsoft documents that ad hoc queries like WHERE Salary > 99999 AND Salary < 100001 infer a masked value. Add row-level security, least privilege and audit.
• Key shipped with the copy or reused across environments: pseudonyms can be recomputed.
• Free-text columns hide names and numbers; null them or run entity detection (Presidio).
• Shifted full dates do not meet HIPAA Safe Harbor, which lists all date elements except year (45 CFR 164.514(b)(2)); they need expert determination.

Variants and alternatives

In production

• Payments and retail: card displays with BIN plus last four (PCI SSC FAQ 1492); US receipts print at most the last 5 digits (15 U.S.C. 1681c(g)).
• Healthcare: HIPAA Safe Harbor de-identification removes listed identifiers, including all date elements except year, before sharing (45 CFR 164.514(b)(2)).
• Data platforms: SQL Server/Azure SQL and Snowflake rewrite columns by role at read time; PostgreSQL Anonymizer does static and dynamic masking.
• Software operations: log redaction as OWASP's Logging Cheat Sheet advises.

Implementations

Sources

• Microsoft, Dynamic data masking, SQL Server docs. docs
• Snowflake, Using Dynamic Data Masking. docs
• PCI SSC, FAQ 1492: PAN masking and truncation with 8-digit BINs. faq
• PostgreSQL Anonymizer docs. docs
• HashiCorp, Transform secrets engine. docs
• Sweeney, Simple Demographics Often Identify People Uniquely, 2000. pdf
• Rocher, Hendrickx, de Montjoye, Estimating the success of re-identifications in incomplete datasets using generative models, Nat. Commun. 2019. doi
• OWASP, Logging Cheat Sheet. docs
• US HHS, 45 CFR 164.514. ecfr
• US Code, 15 U.S.C. 1681c(g), truncation of card numbers on receipts (FACTA). law