Containers and Kubernetes

Deep dive

Options from small to large

Decision table

Code

Minimal production-shaped Deployment: requests, limits, probes, graceful shutdown, non-root.

apiVersion: apps/v1
kind: Deployment
metadata: { name: api }
spec:
  replicas: 3
  strategy: { rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - { maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway,
            labelSelector: { matchLabels: { app: api } } }
      securityContext: { runAsNonRoot: true, runAsUser: 10001 }
      containers:
        - name: api
          image: registry.example.com/api@sha256:3f1c...   # pin by digest
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits:   { memory: "512Mi" }                    # no CPU limit: avoids CFS throttling
          readinessProbe: { httpGet: { path: /ready, port: 8080 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /live,  port: 8080 }, periodSeconds: 10, failureThreshold: 6 }
          lifecycle: { preStop: { exec: { command: ["sleep", "5"] } } }  # let LB drain first
          securityContext: { readOnlyRootFilesystem: true, allowPrivilegeEscalation: false }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec: { minAvailable: 2, selector: { matchLabels: { app: api } } }

Small multi-stage image:

FROM rust:1.82 AS build
WORKDIR /src
COPY . .
RUN cargo build --release --locked
FROM gcr.io/distroless/cc-debian12:nonroot
COPY --from=build /src/target/release/api /api
ENTRYPOINT ["/api"]

Efficiency

• Cost: a container start is a process start plus an image pull, bound by registry bandwidth and layer size; the kubelet skips the pull when the image is already on the node and the policy is IfNotPresent (Kubernetes images docs). A managed control plane is a fixed fee per cluster-hour, nodes bill per hour. A CPU limit is enforced by throttling when the container approaches it (Kubernetes resource docs).
• Levers:
  • small multi-stage images (distroless) — less to pull, target 10-200 MB (order of) — every rollout
  • pin by digest, keep IfNotPresent — node-cached images skip the pull (Kubernetes images docs) — nodes that already ran the image
  • CPU request without a CPU limit — avoids limit-based throttling (Kubernetes resource docs) — latency-sensitive services
  • accurate requests — nodes are bin-packed on requests, so fewer nodes (order of) — shared clusters
  • scale to zero on Cloud Run-style platforms — no cost while idle (Key facts) — spiky HTTP
• Naive → standard → state of the art:
  • latest full-OS images, no requests · slow pulls, idle nodes
  • distroless image by digest, requests set, rolling update with probes · small pulls, packed nodes
  • node autoscaler adding right-sized nodes, scale to zero for spiky services · pay for used capacity (order of)

Implementation notes

• Set memory request = limit (no overcommit) and no CPU limit, only a CPU request; CPU limits cause CFS throttling and tail latency.
• Readiness ≠ liveness: readiness gates traffic; liveness restarts. A liveness probe that checks the DB restarts every pod when the DB blips.
• Handle SIGTERM: stop accepting, finish in-flight requests, exit. The preStop sleep covers the delay before endpoints are removed from load balancers.
• Use a container-native load balancer or Gateway API; avoid one cloud LB per Service (each costs order of $16-20/month on AWS).
• Pin images by digest, scan them, sign them; see Supply chain security.

Pitfalls

• Cross-AZ traffic between pods is billed on AWS/Azure/GCP; chatty services spread over 3 zones pay per GB.
• latest tags and mutable images make rollbacks lie.
• JVM/Go/.NET runtimes reading host CPU count instead of the cgroup quota (old runtimes) spawn too many threads.
• DNS (CoreDNS) and conntrack table exhaustion under high connection churn; use NodeLocal DNSCache and keep-alive pools.
• Upgrading the cluster every ~4 months is mandatory (k8s supports ~14 months per minor); extended support costs extra on EKS.

Implementations

Sources

• Brendan Burns et al., Borg, Omega, and Kubernetes, ACM Queue, 2016. paper
• Abhishek Verma et al., Large-scale cluster management at Google with Borg, EuroSys 2015. paper
• Kubernetes documentation, Pod Lifecycle docs and Resource Management for Pods and Containers docs