Cgroups and namespaces

Deep dive

How it works

In: a command, a set of CLONE_NEW* flags, an id map, and per-controller limits written to files · Out: a process tree with private views, accounted and capped resources, and event counters · State: kernel objects: one namespace instance per kind, one cgroup directory per group (counters and limits as files) · Cost: O(1) syscalls to create; O(tree depth) per memory charge

1. Create namespaces: clone(2) with CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | …, or unshare(2) for the caller; setns(2) joins an existing one through a /proc/<pid>/ns/* file. Creating most kinds needs CAP_SYS_ADMIN; a user namespace needs none and gives the creator full capabilities inside it.
2. Map ids: write inside outside count lines to /proc/<pid>/uid_map and gid_map, once. Unprivileged: one line mapping the caller's own uid. An outside id with no mapping reads as the overflow id 65534; at most 340 lines (since Linux 4.16).
3. Make a cgroup: mkdir under /sys/fs/cgroup; enable controllers in the parent's cgroup.subtree_control. A non-root cgroup holding processes cannot hand controllers to children (no-internal-process rule).
4. Set limits: cpu.max (quota period in µs, default max 100000), cpu.weight (default 100), memory.max/memory.high/memory.low, io.max, pids.max.
5. Join: write the pid to cgroup.procs; children stay in the group.
6. Charge: memory is charged to the cgroup and all ancestors. At a hard limit the kernel reclaims, then runs the OOM killer inside that group.
7. Enforce CPU: per period the group runs at most quota µs summed over its threads, then is throttled until the refill (cpu.stat: nr_periods, nr_throttled, throttled_usec).
8. Observe and control: memory.current, memory.events, *.pressure (PSI); cgroup.freeze stops the tree; cgroup.kill SIGKILLs it.

The real interface:

mkdir /sys/fs/cgroup/demo && cd /sys/fs/cgroup/demo        # as root, cgroup v2 mounted
echo "50000 100000" > cpu.max       # half a CPU: 50 ms per 100 ms
echo 256M > memory.max              # hard limit; OOM inside the group
echo $$ > cgroup.procs              # this shell (and its children) now count here
unshare --user --map-root-user --pid --fork --mount-proc sh   # no root needed

Code

A model of the kernel rules (sources below), std-only Rust; it does not call the kernel. It covers hierarchical memory charging with memory.max/memory.high and memory.events, cpu.max bandwidth with cpu.stat counters, and uid_map translation.

// Model of cgroup v2 rules; it does not call the kernel.
const OVERFLOW_ID: u32 = 65534; // /proc/sys/kernel/overflowuid default

#[derive(Default)]
struct Cg { parent: Option<usize>, max: Option<u64>, high: Option<u64>, current: u64, high_ev: u64, max_ev: u64, oom_ev: u64 }

struct Tree(Vec<Cg>);
impl Tree {
    fn add(&mut self, parent: Option<usize>, max: Option<u64>, high: Option<u64>) -> usize {
        self.0.push(Cg { parent, max, high, ..Default::default() });
        self.0.len() - 1
    }
    /// Charge `n` bytes to `id` and all ancestors; a memory.max at any level refuses it.
    fn charge(&mut self, id: usize, n: u64) -> Result<(), usize> {
        let mut c = Some(id);
        while let Some(i) = c {                                     // pass 1: check every level
            if self.0[i].max.is_some_and(|m| self.0[i].current + n > m) {
                self.0[i].max_ev += 1;                              // memory.events: max, then oom
                self.0[i].oom_ev += 1;                              // (reclaim not modelled)
                return Err(i);
            }
            c = self.0[i].parent;
        }
        c = Some(id);
        while let Some(i) = c {                                     // pass 2: commit
            self.0[i].current += n;
            if self.0[i].high.is_some_and(|h| self.0[i].current > h) { self.0[i].high_ev += 1; } // throttle only
            c = self.0[i].parent;
        }
        Ok(())
    }
    fn uncharge(&mut self, id: usize, n: u64) {
        let mut c = Some(id);
        while let Some(i) = c { self.0[i].current -= n; c = self.0[i].parent; }
    }
}

/// cpu.max "quota period" in usec: CPU time per period, summed over threads.
#[derive(Default, Debug, PartialEq)]
struct CpuStat { nr_periods: u64, nr_throttled: u64, throttled_usec: u64 }

/// `threads` run in parallel and want `demand[k]` CPU-usec in period k; the rest carries over.
fn run_bandwidth(quota: u64, period: u64, threads: u64, demand: &[u64]) -> CpuStat {
    let (mut s, mut backlog) = (CpuStat::default(), 0);
    for d in demand {
        backlog += d;
        s.nr_periods += 1;
        let run = backlog.min(quota);
        backlog -= run;
        if backlog > 0 {                                  // quota gone before the period ended
            s.nr_throttled += 1;
            s.throttled_usec += period - run / threads;   // all threads wait until the next refill
        }
    }
    s
}

/// Wall time of one request of `work` CPU-usec starting at a period start.
fn latency(work: u64, quota: u64, period: u64, threads: u64) -> u64 {
    if work <= quota { return work / threads; }
    let full = (work - 1) / quota;                        // quota-exhausting periods before the last one
    full * period + (work - full * quota) / threads
}

/// uid_map: lines of (inside start, outside start, length); at most 340.
struct IdMap(Vec<(u32, u32, u32)>);
impl IdMap {
    fn new(lines: Vec<(u32, u32, u32)>) -> Option<IdMap> {
        let overlap = |a: (u32, u32, u32), b: (u32, u32, u32)| {
            (a.0 < b.0 + b.2 && b.0 < a.0 + a.2) || (a.1 < b.1 + b.2 && b.1 < a.1 + a.2)
        };
        let bad = lines.len() > 340 || lines.iter().any(|l| l.2 == 0 || l.0.checked_add(l.2).is_none() || l.1.checked_add(l.2).is_none())
            || lines.iter().enumerate().any(|(i, a)| lines[i + 1..].iter().any(|b| overlap(*a, *b)));
        if bad { None } else { Some(IdMap(lines)) }
    }
    fn to_outside(&self, id: u32) -> Option<u32> {
        self.0.iter().find(|l| id >= l.0 && id - l.0 < l.2).map(|l| l.1 + (id - l.0))
    }
    fn to_inside(&self, id: u32) -> Option<u32> {
        self.0.iter().find(|l| id >= l.1 && id - l.1 < l.2).map(|l| l.0 + (id - l.1))
    }
    fn show(&self, outside: u32) -> u32 { self.to_inside(outside).unwrap_or(OVERFLOW_ID) }
}

fn main() {
    let mut t = Tree(Vec::new());
    let root = t.add(None, None, None);
    let pod = t.add(Some(root), Some(256 << 20), None);             // parent: memory.max = 256M
    let a = t.add(Some(pod), Some(200 << 20), Some(128 << 20));     // child: max 200M, high 128M
    let b = t.add(Some(pod), None, None);                           // child without a limit
    assert!(t.charge(a, 150 << 20).is_ok());
    assert_eq!((t.0[a].high_ev, t.0[pod].current), (1, 150 << 20)); // above high: throttled, charge still succeeds
    assert_eq!(t.charge(b, 120 << 20), Err(pod));                   // fits b, but would exceed the parent's 256M
    assert_eq!((t.0[pod].oom_ev, t.0[b].current, t.0[pod].current), (1, 0, 150 << 20)); // failed charge leaves no trace
    t.uncharge(a, 150 << 20);
    assert!(t.charge(b, 120 << 20).is_ok());

    let (q, p) = (50_000, 100_000);                                 // cpu.max "50000 100000" = half a CPU
    let s = run_bandwidth(q, p, 4, &[40_000, 90_000, 20_000, 10_000]);
    assert_eq!(s, CpuStat { nr_periods: 4, nr_throttled: 2, throttled_usec: 175_000 });
    assert_eq!(latency(10_000, q, p, 1), 10_000);                   // fits in the quota
    assert_eq!(latency(60_000, q, p, 1), 110_000);                  // 10 ms spills past a refill: 110 ms wall
    assert_eq!(latency(200_000, q, p, 4), 312_500);                 // 4 threads burn 50 ms in 12.5 ms, then wait

    let m = IdMap::new(vec![(0, 1000, 1), (1, 100_000, 65_535)]).unwrap(); // rootless: inside 0 = outside 1000
    assert_eq!((m.to_outside(0), m.to_outside(5), m.to_outside(70_000)), (Some(1000), Some(100_004), None));
    assert_eq!((m.show(1000), m.show(100_004), m.show(0)), (0, 5, OVERFLOW_ID)); // unmapped outside uid shows as 65534
    assert!(IdMap::new(vec![(0, 1000, 10), (5, 2000, 10)]).is_none()); // overlapping inside ranges are rejected
    println!("ok");
}

Measured on this Mac (Apple M4 Pro), three runs, release build: one charge plus uncharge on a 5-level chain took 7.0 to 7.1 ns. That times the model, not Linux.

Complexity and constants

• Memory charge: O(depth) counter updates. Throttling granularity is the period: default 100 ms; quota and period at least 1 ms, period at most 1 s; quota is handed to CPUs in slices (default 5 ms).
• A request of w CPU-µs under quota q per period p needs ⌈w/q⌉-1 whole periods plus the remainder: 60 ms of work at 50 ms per 100 ms takes 110 ms (model above).

Efficiency

• Cost: setup (clone, id map, mounts, network) dominates; steady state is counter updates per allocation and scheduler tick. Throttling costs latency, not throughput. No kernel figure measured here (model: 7 ns per charge pair, above). Bound: the period.
• Levers:
  • cpu.weight instead of cpu.max — shares CPU only under contention, no hard stalls (Go blog) — when no hard cap is required.
  • cpu.max.burst — banks unused quota for spikes (cgroup v2 guide; default 0) — bursty latency-sensitive services.
  • Container-aware runtimes — Go 1.25 sets GOMAXPROCS from the CPU limit to avoid throttling (Go blog) — any runtime sizing thread pools by CPU count.
  • memory.high plus a monitor — throttles instead of killing (cgroup v2 guide) — slowable jobs.
  • cpuset pinning (CPU isolation) — dedicated cores, no quota — latency-critical work.
• Naive → standard → state of the art: one VM per tenant · namespaces plus cgroups via an OCI runtime · same with user namespaces, seccomp and dropped capabilities; microVMs for hostile code.

Implementation notes

• Enable controllers top-down: +cpu +memory in each ancestor's cgroup.subtree_control. Delete a cgroup only when empty (cgroup.kill first).
• PID namespace: the first process is PID 1; only signals it has a handler for reach it from inside, and when it dies the kernel SIGKILLs the namespace. With unshare --pid, use --fork.
• Runtime order: user ns, id maps, mounts, drop capabilities, seccomp filter, exec.
• Read pressure, not usage: memory.pressure (PSI some/full) shows whether a group stalls; memory.current includes reclaimable cache (memory pressure).

Pitfalls

• Quota on a many-threaded runtime: threads burn the quota in a few ms, then the group stalls until the refill; tail latency grows by up to a period (Go blog). Use fewer threads, a larger quota or weights.
• Runtimes sizing pools from the host CPU count over-subscribe the quota (Go before 1.25); use container-aware versions.
• In-group OOM: memory.events oom_kill counts victims; memory.oom.group=1 kills the whole group together.
• An app as PID 1 in a container ignores SIGTERM without a handler and leaves zombies: put an init such as tini in front (process supervision).
• Namespaces are not a VM boundary: the kernel is shared, so a kernel bug crosses it. Add seccomp, capability drops and other sandboxing.
• v1 and v2 can run side by side, but a controller can be in only one of them; v2 has fewer controllers.

Variants and alternatives

In production

• Container runtimes (runc, crun, youki); systemd maps CPUQuota=, MemoryMax= and TasksMax= to cpu.max, memory.max and pids.max.
• bubblewrap sandboxes unprivileged programs (Flatpak); Slurm's cgroup plugin constrains cores and RAM per job.

Implementations

Sources

• Linux kernel docs, Control Group v2. docs
• Linux kernel docs, CFS Bandwidth Control. docs
• Linux kernel docs, PSI - Pressure Stall Information. docs
• namespaces(7), user_namespaces(7), cgroups(7), pid_namespaces(7), Linux man-pages. namespaces · user · cgroups · pid
• unshare(1) man; systemd.resource-control(5) man
• Go team, Container-aware GOMAXPROCS, 2025. blog