VMs and bare metal

Deep dive

Instance families (AWS names; GCP/Azure have equivalents)

Price orders of magnitude (2026-ish)

Decision table

Code

Cloud-init for a hardened, tuned server (works on Hetzner, AWS, GCP, Azure):

#cloud-config
package_update: true
packages: [nftables, chrony, unattended-upgrades]
users:
  - name: ops
    groups: sudo
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys: ["ssh-ed25519 AAAA... ops@laptop"]
ssh_pwauth: false
disable_root: true
write_files:
  - path: /etc/sysctl.d/90-server.conf
    content: |
      net.core.somaxconn = 65535
      net.ipv4.tcp_max_syn_backlog = 65535
      net.ipv4.ip_local_port_range = 1024 65535
      net.core.rmem_max = 67108864
      net.core.wmem_max = 67108864
      net.ipv4.tcp_congestion_control = bbr
      vm.swappiness = 1
  - path: /etc/security/limits.d/90-nofile.conf
    content: "* - nofile 1048576\n"
runcmd:
  - sysctl --system
  - systemctl enable --now nftables chrony

Check what you actually got:

nproc; lscpu | grep -E 'Model name|Thread|NUMA'
cat /sys/block/nvme0n1/queue/scheduler          # want [none] for NVMe
fio --name=r --rw=randread --bs=4k --iodepth=64 --direct=1 --size=4G --runtime=30 --ioengine=io_uring

Efficiency

• Cost: a gp3 volume includes 3,000 IOPS and 125 MiB/s, can be provisioned to 80,000 IOPS and 2,000 MiB/s, and has single-digit ms latency (EBS docs, as of 2026-10); local NVMe is order of 500k-1M+ IOPS at about 0.1 ms. Spot Instances are up to 90 % cheaper, with a two-minute interruption notice (AWS Spot page and EC2 User Guide). Bound: IOPS and latency for databases, vCPU price for compute.
• Levers:
  • local NVMe instead of a network volume — order of 100x the IOPS (Key facts) — databases, Kafka, search
  • provision gp3 IOPS and throughput independent of size — up to 80,000 and 2,000 MiB/s (EBS docs) — I/O-bound services on network disks
  • Spot for stateless work — up to 90 % off (AWS Spot page) — CI, batch
  • Arm instances — order of 20 % cheaper per vCPU (Key facts) — code that rebuilds for arm64
  • measure with fio using io_uring and O_DIRECT — know the real device limit before sizing (Code above) — every new box
• Naive → standard → state of the art:
  • burstable VM, default volume, on demand · 3,000 IOPS, throttles when credits end
  • right family, provisioned volume, commitments, Spot for batch · IOPS and price match the load
  • bare metal with local NVMe, pinned threads, private network · full device IOPS, no CPU steal

Implementation notes

• Use ioengine=io_uring or libaio and O_DIRECT for databases on NVMe; see io_uring.
• Local NVMe on cloud VMs is ephemeral: stop/terminate wipes it. Replicate or back up continuously.
• Bare metal: order RAID1 for the OS, monitor SMART, keep one spare server or a cloud fallback; disk failure is a when, not if.
• Set CPU governor to performance and pin hot threads on latency-critical boxes; check steal in top on VMs.
• Hetzner/OVH private networks (vSwitch, vRack) keep server-to-server traffic off the public link and free.

Pitfalls

• Burstable (t-family, B-series, e2-micro/shared core) instances throttle to a fraction of a core once credits are spent; load tests pass, production stalls.
• gp2/gp3 baseline IOPS and throughput are per volume; a database on one default volume caps near 3k IOPS / 125 MB/s.
• Spot instances in one instance type and one AZ get reclaimed together; diversify types and zones.
• Bandwidth on a "1 Gbit/s" server is shared at the switch on some budget offers; measure with iperf3.
• Forgetting IPv4 cost and stopped-but-not-terminated volumes, which keep billing.

Implementations

Sources

• AWS, Amazon EC2 Instance Types docs and EBS volume types docs (documentation); Amazon EC2 Spot Instances page
• Hetzner Online, Cloud and Dedicated Root Server price lists (web pages). cloud · dedicated
• Brendan Gregg, Systems Performance, 2nd ed., Addison-Wesley, 2020. book