Secrets management

Deep dive

Decision table

Code

SOPS with age (secrets encrypted in git, values only, keys readable):

age-keygen -o ~/.config/sops/age/keys.txt         # private key stays local / in CI secret store
cat > .sops.yaml <<'EOF'
creation_rules:
  - path_regex: secrets/prod\..*\.yaml$
    age: age1...   # your public key from age-keygen
EOF
sops --encrypt --in-place secrets/prod.app.yaml
sops exec-env secrets/prod.app.yaml './server'    # decrypted only into the child process env

GitHub Actions to AWS without stored keys:

permissions:
  id-token: write   # allow requesting the OIDC token
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4   # pin to a commit SHA in real use
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-prod
          aws-region: eu-central-1

The role's trust policy must restrict token.actions.githubusercontent.com:sub to repo:org/repo:ref:refs/heads/main (or an environment), not just the org.

Rust: load from a mounted file, keep it out of Debug output (secrecy = "0.10"):

use secrecy::{ExposeSecret, SecretString};

#[derive(Debug)] // SecretString's Debug prints [REDACTED]
pub struct Config { pub db_password: SecretString }

pub fn load() -> std::io::Result<Config> {
    // e.g. /run/secrets/db_password from Docker/K8s/systemd LoadCredential=
    let dir = std::env::var("CREDENTIALS_DIRECTORY").unwrap_or_else(|_| "/run/secrets".into());
    let raw = std::fs::read_to_string(format!("{dir}/db_password"))?;
    Ok(Config { db_password: SecretString::from(raw.trim_end().to_owned()) })
}

pub fn connect_url(c: &Config) -> String {
    // expose only at the point of use; SecretString zeroizes on drop
    format!("postgres://app:{}@db:5432/app", c.db_password.expose_secret())
}

Efficiency

Cost: dominated by network round trips to the store and by operator effort, not CPU. The bound is latency and API rate limits.

Levers:

• Fetch once — read at startup (or a mounted file) and keep in memory; per-request fetches hit rate limits — all services.
• Envelope encryption — bulk data encrypted locally, only the small data key goes to the KMS; cache decrypted data keys briefly — encrypted data at rest.
• Workload identity and OIDC federation — nothing stored, so nothing to rotate or leak — cloud workloads and CI.
• Scan the diff — gitleaks on staged or pushed changes takes milliseconds to seconds (order of); scan full history once — repositories.
• Dynamic credentials — Vault or OpenBao add one API call per lease but make leaks self-expiring — high-value systems.

Naive → standard → state of the art:

• Naive: secrets in code or env files, fetched per request · leaks, rate limits
• Standard: SOPS with age, no service, no API calls · zero runtime cost
• State of the art: workload identity, envelope encryption, dynamic leases · one API call per lease or key

Rotation

1. Support two valid secrets at once (two DB users, two API keys, JWKS with two kids).
2. Create new → deploy consumers reading the new one → revoke old → alert if old is still used.
3. Automate on a schedule (order of 30-90 days for static secrets) so the emergency rotation path is the tested one.

Detection and response

• Push protection and pre-commit hooks stop most leaks before they land; scan full history once when adopting.
• Canary tokens (fake AWS keys, fake DB rows) alert when someone uses stolen material.
• On a leak: revoke/rotate first, then audit the provider's logs for use, then clean history. Assume bots used it within minutes.

Pitfalls

• Secrets in git history: deleting the file doesn't remove them; rotate, then rewrite history if needed.
• Environment variables leak: /proc/<pid>/environ, crash reporters, child processes, docker inspect, debug pages. Prefer files on tmpfs or fetching from the store.
• Docker build args and COPY .env: baked into image layers forever. Use BuildKit --secret.
• Logging config structs, HTTP clients logging Authorization headers, error messages with connection strings.
• One shared "admin" key across services: no attribution, impossible to rotate safely.
• CI secrets exposed to pull requests from forks (pull_request_target workflows that check out PR code).
• Frontend bundles or mobile apps with "secret" keys: anything shipped to clients is public.
• Long-lived cloud access keys for humans: use SSO with short sessions.

In production

• Software delivery: GitHub Actions requests short-lived cloud credentials through OIDC instead of stored keys.
• Container platforms: Kubernetes encrypts Secret objects in etcd via a KMS provider (DEK per API server in kms v2).
• IoT and edge: the AWS IoT Greengrass secret manager component deploys secrets from AWS Secrets Manager to core devices and keeps them encrypted there under the device's private key.

Implementations

Sources

• OWASP, Secrets Management Cheat Sheet. doc
• Barker, NIST SP 800-57 Part 1 Rev. 5: Recommendation for Key Management, 2020. doc
• SOPS project documentation (getsops/sops). code
• age specification (C2SP). spec
• GitHub Docs, Configuring OpenID Connect in Amazon Web Services. doc
• HashiCorp, Vault documentation: dynamic secrets. doc · code
• Kubernetes, Encrypting Confidential Data at Rest. docs
• AWS, Secret manager component, AWS IoT Greengrass Developer Guide. docs