Interrupt handling

Deep dive

How it works

In: a device or timer signal on line/vector N · Out: the registered handler runs once on one CPU, then the interrupted code resumes · State: per-line pending, mask and priority bits in the controller; vector table; per-CPU bottom-half queues · Cost: entry/exit (not measured here) plus handler time; rate limited by coalescing.

1. Signal: a wire (edge or level) or an MSI/MSI-X write to a controller-owned address. The controller sets pending for that vector.
2. Route: x86 has an I/O APIC for pin interrupts and a local APIC per core; Arm GICv3 has a distributor, per-core redistributors and CPU interfaces (the kernel's VGICv3 doc names distributor and redistributor). Priority and mask decide which pending line is delivered, and to which CPU.
3. Vector: the CPU saves minimal state and indexes a table: the IDT on x86, the exception vector table at VBAR_EL1 on Armv8-A, the NVIC vector table on Cortex-M (these Arm details: not verified here).
4. Top half (hard IRQ): acknowledge the device, read what happened, mask the line if work is deferred, queue a bottom half. Linux: request_irq(); the primary handler returns IRQ_WAKE_THREAD to run thread_fn of request_threaded_irq() (genericirq).
5. EOI: tell the controller the line is serviced (irq_eoi/irq_ack in the flow handlers); lower-priority lines wait until then.
6. Bottom half: softirq, tasklet, workqueue or threaded handler, with interrupts enabled (kernel-hacking). Networking runs NAPI: poll up to budget; if work remains, stay in polling, else unmask (NAPI).
7. Return: restore state, resume the interrupted code.
8. Steer: /proc/irq/N/smp_affinity picks CPUs; irqbalance may override manual settings (affinity, scaling).

Code

A model, not a kernel: a priority controller with pending/mask bits, a top half that only masks and schedules, a NAPI-style budgeted poll that re-enables the line when the queue empties, and device-side coalescing (raise after n events or after timeout ticks). Discrete ticks, no nesting.

use std::collections::VecDeque;

/// Interrupt controller: one pending and one mask bit per line; line number = priority (0 highest).
#[derive(Default)]
struct Ctl { pending: u32, masked: u32 }
impl Ctl {
    fn raise(&mut self, line: u32) { self.pending |= 1 << line; }   // edge: repeated raises merge
    fn mask(&mut self, line: u32) { self.masked |= 1 << line; }
    fn unmask(&mut self, line: u32) { self.masked &= !(1 << line); }
    /// Acknowledge: pick the highest-priority unmasked pending line and clear it.
    fn ack(&mut self) -> Option<u32> {
        let ready = self.pending & !self.masked;
        if ready == 0 { return None; }
        let line = ready.trailing_zeros();
        self.pending &= !(1 << line);
        Some(line)
    }
}

const NIC: u32 = 3;
struct Cfg { n: usize, timeout: u64, budget: usize } // coalesce after n events or timeout ticks

#[derive(Default, Debug)]
struct Stats { irqs: u64, handled: u64, max_latency: u64 }

/// One NIC line. `arrivals[t]` = events arriving at tick t. The CPU retires `budget` events per tick.
fn run(arrivals: &[usize], cfg: &Cfg) -> Stats {
    let (mut ctl, mut st) = (Ctl::default(), Stats::default());
    let mut queue: VecDeque<u64> = VecDeque::new();   // arrival tick of each queued event (device ring)
    let (mut since, mut oldest, mut polling) = (0usize, None::<u64>, false);
    let mut t = 0u64;
    while (t as usize) < arrivals.len() || !queue.is_empty() {
        let k = arrivals.get(t as usize).copied().unwrap_or(0);
        for _ in 0..k { queue.push_back(t); since += 1; oldest.get_or_insert(t); }
        // device-side coalescing: raise after n events, or when the oldest has waited `timeout`
        if since >= cfg.n || oldest.is_some_and(|o| t - o >= cfg.timeout) {
            ctl.raise(NIC); since = 0; oldest = None;
        }
        // CPU: hard IRQ (top half) only masks the line and schedules the bottom half
        while let Some(line) = ctl.ack() {
            assert_eq!(line, NIC);
            st.irqs += 1; ctl.mask(NIC); polling = true;
        }
        if polling { // bottom half: NAPI-style poll with a budget
            for _ in 0..cfg.budget {
                match queue.pop_front() {
                    Some(a) => { st.handled += 1; st.max_latency = st.max_latency.max(t - a); }
                    None => break,
                }
            }
            if queue.is_empty() { polling = false; ctl.unmask(NIC); } // done: back to interrupt mode
        }
        t += 1;
    }
    st
}

fn main() {
    // 1. masked line stays pending until unmasked
    let mut c = Ctl::default();
    c.raise(5); c.mask(5);
    assert_eq!(c.ack(), None);
    c.unmask(5);
    assert_eq!(c.ack(), Some(5));
    // 2. priority order: lowest number first, regardless of arrival order
    c.raise(7); c.raise(2); c.raise(5);
    assert_eq!((c.ack(), c.ack(), c.ack(), c.ack()), (Some(2), Some(5), Some(7), None));
    // 3. burst: 4 events/tick for 1000 ticks, CPU retires 2/tick; one IRQ per event would be 4000
    let burst = vec![4usize; 1000];
    let s = run(&burst, &Cfg { n: 1, timeout: 1, budget: 2 });
    assert_eq!(s.handled, 4000);
    assert!(s.irqs < 10, "NAPI path: {} IRQs for 4000 events", s.irqs);
    let burst_irqs = s.irqs;
    // 4. coalescing bounds latency: 1 event per 10 ticks, raise at 8 events or 50 ticks
    let sparse: Vec<usize> = (0..2000).map(|t| (t % 10 == 0) as usize).collect();
    let s = run(&sparse, &Cfg { n: 8, timeout: 50, budget: 64 });
    assert!(s.max_latency <= 50, "latency {} > timeout", s.max_latency);
    assert!(s.irqs <= 2000 / 50 + 1);
    println!("burst: {} IRQs for 4000 events; coalescing: {} IRQs, max latency {} ticks", burst_irqs, s.irqs, s.max_latency);
    // timing of the MODEL itself (not of any kernel): events/s through run()
    let big = vec![8usize; 1_000_000];
    let t0 = std::time::Instant::now();
    let s = run(&big, &Cfg { n: 64, timeout: 100, budget: 64 });
    let dt = t0.elapsed().as_secs_f64();
    println!("{} events, {} IRQs, {:.1} M events/s", s.handled, s.irqs, s.handled as f64 / dt / 1e6);
}

Complexity and constants

Per interrupt O(1) in the controller (lowest set bit). The cost is elsewhere: pipeline flush, stack switch, register save, cache and TLB pollution (not measured here). Bottom-half cost scales with events handled. Measured on this Mac (Apple M4 Pro), timing the model only (no kernel, no real interrupt): about 6.1-6.6 x 10^8 events/s over three runs (8 M events, coalescing 64, budget 64). It says nothing about real interrupt cost.

Efficiency

• Cost: one hard IRQ is a fixed entry/exit cost plus the handler (not measured here); it is paid per interrupt, so at high event rates the interrupt rate, not the work, is the problem. Bound: entry latency and cache disruption, not bandwidth.
• Levers:
  • Device coalescing (ethtool -C: rx-usecs, rx-frames, adaptive-rx (ethtool(8))) — fewer IRQs per event, bounded by the timeout the model asserts — when added latency of up to rx-usecs is acceptable.
  • NAPI polling under load — the line stays masked while the queue is non-empty (model: 1 IRQ for 4000 events) — high, steady rates.
  • napi_defer_hard_irqs plus gro_flush_timeout — defers re-enabling IRQs after empty polls (NAPI) — busy-poll servers that accept some idle-time latency.
  • One MSI-X vector per queue, affinity per core — no cross-core cache bouncing for that queue — multi-queue NICs and NVMe.
  • Poll-mode drivers — no interrupts except link status change (DPDK PMD) — dedicated cores, line-rate traffic.
• Naive → standard → state of the art: interrupt per event, handler does everything · livelock under load (Mogul & Ramakrishnan); top/bottom half + NAPI + coalescing · bounded interrupt rate; polling cores (DPDK), or IRQ suspension with busy polling (NAPI) · no interrupt cost on the hot path.

Implementation notes

• Mask the one line, not all interrupts on the CPU: other devices keep running.
• Level-triggered lines re-fire if the device still asserts: quiet the device, then EOI/unmask.
• Pin-based lines can be shared, so every handler must check its own device; MSI is never shared (MSI HOWTO). Allocate with pci_alloc_irq_vectors().
• PREEMPT_RT forces threaded handlers: the primary handler only wakes the thread, the source stays masked until it finishes (kernel RT theory).

Pitfalls

• Interrupt storm: a level line never cleared, or a handler that forgets to ack, pins a CPU at 100 % in IRQ context.
• Spurious interrupts: handlers that assume work exists spin or corrupt state; re-check the device.
• Sleeping or slow work in a hard-IRQ handler; use IRQ-safe locks only.
• Lost wakeup: unmask before re-checking the queue and an event arriving in between waits for the next interrupt (check once more after unmasking, as NAPI's napi_complete_done() protocol is built to allow).
• Coalescing timers set too high hurt tail latency; irqbalance can undo manual affinity.

Variants and alternatives

In production

• Kernels (every driver), Cortex-M firmware via the NVIC (CMSIS-Core; SPI/I2C drivers), network servers (coalescing, affinity; DPDK forwarders poll).

Implementations

Sources

• Mogul, Ramakrishnan, Eliminating Receive Livelock in an Interrupt-driven Kernel, USENIX ATC 1996. paper
• Linux kernel docs cited inline (genericirq, MSI HOWTO, NAPI, SMP IRQ affinity, scaling, real-time theory, kernel-hacking); DPDK PMD; ethtool(8).