GPU cloud

Deep dive

Chips (order of, dense BF16)

Provider classes

Decision table

Code

Launch checklist on a fresh GPU box (any provider):

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
nvidia-smi topo -m                                 # NVLink (NV#) vs PCIe (PHB/SYS) between GPUs
nvidia-smi --query-gpu=index,compute_cap,pcie.link.gen.current,pcie.link.width.current --format=csv
# NCCL bandwidth: expect ~200+ GB/s bus bw intra-node on H100 NVLink, far less over PCIe
git clone https://github.com/NVIDIA/nccl-tests && cd nccl-tests && make -j
./build/all_reduce_perf -b 8M -e 1G -f 2 -g $(nvidia-smi -L | wc -l)

Resumable training loop for spot/preemptible GPUs (Rust; the same pattern applies in any framework):

use std::fs::{self, File};
use std::io::Write;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;

/// Anything that can serialize its full state: weights, optimizer, step, data position, RNG.
pub trait Trainer {
    fn step(&mut self) -> f32;                       // one optimizer step, returns the loss
    fn step_count(&self) -> u64;
    fn save(&self) -> Vec<u8>;
    fn load(&mut self, bytes: &[u8]);
}

fn save_atomic(path: &str, bytes: &[u8]) -> std::io::Result<()> {
    let tmp = format!("{path}.tmp");
    let mut f = File::create(&tmp)?;
    f.write_all(bytes)?;
    f.sync_all()?;                                   // data durable before the rename
    fs::rename(&tmp, path)                           // atomic: never a half-written checkpoint
}

pub fn run(t: &mut impl Trainer, ckpt: &str, total: u64, every: u64) -> std::io::Result<()> {
    // preemption ends in an OS shutdown or pod stop, which sends SIGTERM (signal-hook crate)
    let stop = Arc::new(AtomicBool::new(false));
    signal_hook::flag::register(signal_hook::consts::SIGTERM, Arc::clone(&stop))?;
    if let Ok(bytes) = fs::read(ckpt) { t.load(&bytes) }       // resume where we left off
    while t.step_count() < total {
        let loss = t.step();
        let n = t.step_count();
        if n % 100 == 0 { eprintln!("step {n} loss {loss:.4}") }
        let stopping = stop.load(Ordering::Relaxed);
        if n % every == 0 || stopping {
            save_atomic(ckpt, &t.save())?;           // then copy to object storage in the background
            if stopping { break }
        }
    }
    Ok(())
}

On AWS, poll http://169.254.169.254/latest/meta-data/spot/instance-action (IMDSv2 token required) to get the 2-minute notice and checkpoint before the shutdown starts.

Efficiency

• Cost: GPU time bills per hour (per second on pod platforms) whether the GPU computes or waits, so the effective price is the hourly price divided by model FLOPs utilisation (MFU, defined in the PaLM paper). 35-55 % MFU is good for transformer training on H100 (order of). H100 prices are order of $2-3 per hour on neoclouds and $4-8 on hyperscalers. AWS gives a two-minute notice before reclaiming a Spot Instance (EC2 User Guide). Bound: interconnect and data loading as much as FLOPs.
• Levers:
  • local NVMe or same-datacenter dataset with prefetch — raises MFU; below 25 % look at data loading first (order of) — all training
  • NVLink inside a node, InfiniBand or EFA across nodes; verify with nccl-tests — collective bandwidth decides scaling (nccl-tests) — multi-GPU jobs
  • Spot or marketplace GPUs with atomic checkpoints — up to 90 % off on AWS Spot (AWS Spot page) — restartable training
  • idle shutdown — idle GPUs bill in full (order of) — notebooks, pods waiting on data
  • reservations — 30-60 % off at high utilisation (order of) — steady serving and long pretraining
• Naive → standard → state of the art:
  • on-demand hyperscaler GPUs, data pulled over the WAN · highest price, low MFU
  • neocloud GPUs, data on local NVMe, periodic checkpoints · 2-3x cheaper per hour (order of)
  • Spot with resumable training, reserved cluster for long pretraining · up to 90 % off Spot (AWS), MFU 35-55 %

Implementation notes

• Measure MFU (model FLOPs utilisation): 35-55 % is good for transformer training on H100; below 25 % look at data loading, small batches or communication.
• Keep the dataset on local NVMe or a parallel filesystem in the same datacenter; stream from object storage with large sequential reads and prefetch.
• Sync checkpoints to object storage asynchronously; a lost node must cost minutes, not hours.
• Check driver/CUDA/NCCL versions on day one; mismatches are the top cause of "it was slow on that provider".
• Reservations: hyperscalers and neoclouds sell 1-week to 3-year blocks; price a reservation vs on-demand at your expected utilisation.

Pitfalls

• Paying for idle GPUs: notebooks left running, pods waiting on data. Add idle shutdown.
• PCIe-only 8-GPU boxes sold as "8x H100" (H100 PCIe or no NVLink) run tensor parallelism several times slower.
• Ephemeral container disks: pod restart deletes unsynced checkpoints.
• Marketplace hosts can see your data and weights; don't put secrets or private data there.
• Quota: new hyperscaler accounts often have 0 GPU quota; requests take days.

Implementations

Sources

• NVIDIA, H100 Tensor Core GPU (datasheet and product page). page and Blackwell (B200) datasheet.
• NVIDIA, nccl-tests (GitHub repository). code
• Aakanksha Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022 (MFU definition). paper
• AWS, Amazon EC2 Spot Instances. page