Hot-standby redundancy

Deep dive

How it works

Parts:

• primary node — runs the loop, drives the outputs, sends one heartbeat with state per cycle
• backup node — runs the same loop on the same inputs, adopts each checkpoint, counts silent cycles
• sync link — carries the heartbeat (ideally two independent paths)
• epoch — counter bumped on every takeover; higher epoch wins
• fenced actuator — remembers the highest epoch seen, drops outputs from lower ones

Operations: replicate, heartbeat, detect, take over, fence, resolve duplicates, rejoin (steps below).

1. Replicate execution. Both nodes read the same inputs (wired to both, or mirrored over a sync link) and run the same fixed-rate loop. Only the primary's outputs reach the plant.
2. Heartbeat + checkpoint. Each cycle the primary sends {id, epoch, state}. The backup overwrites its state with it. This corrects any divergence (different rounding, missed cycle, a restarted node) and doubles as the liveness signal.
3. Detect. The backup counts consecutive cycles without a valid heartbeat; at miss_limit it promotes itself. The timeout must exceed worst-case link jitter plus the peer's longest legal pause, or failovers become spurious.
4. Take over with a higher epoch. epoch += 1. The backup already holds current state, so the first output continues the previous one (bumpless transfer) instead of restarting from initial conditions.
5. Fence. The actuator side (output module, valve controller, storage) remembers the highest epoch seen and drops outputs from lower ones. This is the same idea as fencing tokens for a lease.
6. Resolve duplicates. A node that sees a heartbeat with a higher epoch demotes itself and resyncs. On equal epochs (both promoted at once) the lower id wins.
7. Rejoin. A repaired node starts as backup, adopts the next checkpoint, and only then counts as redundancy again. Redundancy is lost until that happens, so alarm on it.

cycle:    ... 49    50   51   52   53
primary:  out  out  X (dead)
backup:   shadow  s=0  s=1  s=2  s=3 -> promote, epoch 1, out (continues the integrator)
actuator:           holds last value     accepts epoch 1; epoch 0 now rejected

Code

Std only; one node, one fenced actuator. Tested by a lockstep harness (below): crash of the primary at cycle 50 gives first backup output at cycle 53, equal to the ideal integral (bumpless); a restarted node resyncs and ends with identical state; a link cut with both nodes alive ends with exactly one primary and the stale primary's outputs rejected.

#[derive(Clone, Copy, PartialEq, Debug)]
pub enum Role { Primary, Backup }

/// Sent by the primary once per cycle: liveness proof and state checkpoint in one message.
#[derive(Clone, Copy, Debug)]
pub struct Heartbeat { pub id: u8, pub epoch: u32, pub state: i64 }

#[derive(Clone, Copy, Debug)]
pub struct Output { pub epoch: u32, pub value: i64 }

pub struct Node {
    pub id: u8,
    pub role: Role,
    pub epoch: u32,
    pub state: i64, // controller state, e.g. an integrator
    silent: u32,    // cycles since the last valid heartbeat
    miss_limit: u32,
}

impl Node {
    pub fn new(id: u8, role: Role, miss_limit: u32) -> Self {
        Node { id, role, epoch: 0, state: 0, silent: 0, miss_limit }
    }

    /// One control cycle. Both nodes read the same input and run the controller;
    /// only the primary's output reaches the plant. `rx` is the heartbeat that
    /// arrived since the last cycle (it carries the sender's state after cycle n-1).
    pub fn tick(&mut self, input: i64, rx: Option<Heartbeat>) -> (Option<Heartbeat>, Option<Output>) {
        let mut valid = false;
        if let Some(hb) = rx {
            let newer = hb.epoch > self.epoch;
            // same epoch: the primary wins over a backup; two primaries: lower id wins
            let tie = hb.epoch == self.epoch
                && (self.role == Role::Backup || hb.id < self.id);
            if newer || tie {
                self.epoch = hb.epoch;
                self.role = Role::Backup; // demote if we were primary
                self.state = hb.state;    // resync to the checkpoint
                valid = true;
            }
        }
        self.state += input; // controller step (integrator)
        if self.role == Role::Backup {
            self.silent = if valid { 0 } else { self.silent + 1 };
            if self.silent >= self.miss_limit {
                self.role = Role::Primary; // takeover: new epoch fences the old primary
                self.epoch += 1;
                self.silent = 0;
            }
        }
        if self.role == Role::Primary {
            let hb = Heartbeat { id: self.id, epoch: self.epoch, state: self.state };
            (Some(hb), Some(Output { epoch: self.epoch, value: self.state }))
        } else {
            (None, None)
        }
    }
}

/// The plant-side gate: outputs from an older epoch are ignored (fencing).
#[derive(Default)]
pub struct Actuator { max_epoch: u32, pub rejected: u32, pub last: Option<Output> }

impl Actuator {
    pub fn drive(&mut self, o: Output) -> bool {
        if o.epoch < self.max_epoch { self.rejected += 1; return false; }
        self.max_epoch = o.epoch;
        self.last = Some(o);
        true
    }
}

Complexity and constants

• Per cycle: one send (primary), one receive and compare (backup), O(state) copy for the checkpoint. Heartbeat above is 13 bytes packed (1 + 4 + 8), 16 in memory.
• Detection: miss_limit * period. VRRP's rule is 3 * advert_interval + skew with skew = (256 - priority) * advert_interval / 256; with the defaults (1 s, priority 100) that is about 3.6 s.
• State size dominates: a 1 kHz loop with a 200-byte checkpoint is 200 kB/s (1.6 Mbit/s), trivial on Ethernet, impossible on a 9.6 kbit/s serial link.
• The plant sees miss_limit missing outputs (3 in the test); the actuator's hold-last-value covers the gap.

Efficiency

• Cost: per cycle one heartbeat send, one receive-and-compare and an O(state) copy; the heartbeat above is 13 bytes packed (16 in memory). A 1 kHz loop with a 200-byte checkpoint is 200 kB/s: trivial on Ethernet, impossible on 9.6 kbit/s serial. Failover time is miss_limit * period; VRRP's rule is 3 * advert_interval + skew, about 3.6 s at defaults (RFC 5798). A hot standby burns the same CPU as the primary. Bound: detection latency and link bandwidth, not compute.
• Levers:
  • send state, not history — checkpoint of integrators and timers is small; input replay costs more — compact state
  • delta checkpoints — changed fields each cycle, full state every N — large state
  • heartbeat piggybacked on the checkpoint — one packet per cycle, no extra syscalls — always
  • batch loops' checkpoints per frame — fewer packets — many loops per node
  • warm or cold standby — saves the second CPU and link, costs a restart transient — tolerant loops
• Naive → standard → state of the art:
  • restart on fault, watchdog only · downtime of a full boot plus initialisation
  • heartbeat with periodic checkpoint, hot standby · about miss_limit cycles
  • dual paths plus actuator fencing · same cycles, no split brain

Implementation notes

• Give the heartbeat a monotonic sequence number as well, so a delayed old packet cannot reset the miss counter or overwrite newer state.
• Use two independent sync paths (dedicated cable plus the plant network) and count a peer as alive if either delivers; this removes most spurious takeovers.
• Fencing must live at the consumer: output cards, drives, or the shared disk. Where hardware cannot check an epoch, cut the old primary off by force (power switch, reset line, STONITH in Pacemaker clusters).
• Input mirroring must be deterministic: both nodes need the same inputs for the same cycle, or the shadow diverges and every checkpoint becomes a small jump on takeover.
• Start the watchdog on both nodes; a hung primary that stops heartbeating is then also reset, and a repaired node rejoins as backup.
• Expose role and epoch as a diagnostic tag: a plant silently running on its backup has no redundancy left.

Pitfalls

• Split brain: both nodes drive outputs after a link failure. Without an actuator-side epoch check or forced shutdown of the other node, you get two writers.
• Takeover from a bad primary: a standby copies a corrupted checkpoint just as faithfully. Validate state (CRC, range checks) before adopting it.
• Preemption flap: a recovered high-priority node that retakes the role immediately causes a second blip. VRRP makes preemption configurable; for plants, prefer manual switch-back.
• Timeout inside the longest legal pause: a flash erase or a GC pause longer than miss_limit * period causes spurious failover.
• Untested failover: the standby path runs rarely, so its rot is invisible. Force a switch-over on a schedule.
• Common cause: same software, power supply or rack defeats redundancy.

Variants and alternatives

In production

• Computing: process pairs in Tandem NonStop systems (checkpointing primary to backup); Pacemaker/Corosync clusters for databases and services.
• Industrial: redundant PLC pairs such as Siemens S7-400H / S7-1500R/H and Rockwell ControlLogix redundancy (see PLC scan cycle and OPC UA server redundancy).
• Networking: VRRP routers (RFC 5798), implemented by keepalived.

Implementations

Sources

• Jim Gray, Why Do Computers Stop and What Can Be Done About It?, Tandem TR 85.7, 1985 (process pairs). PDF
• N. Budhiraja, K. Marzullo, F. B. Schneider, S. Toueg, The Primary-Backup Approach, in Distributed Systems, 2nd ed., ed. S. Mullender, ACM Press/Addison-Wesley, 1993.
• P. A. Alsberg, J. D. Day, A Principle for Resilient Sharing of Distributed Resources, ICSE 1976.
• S. Nadas (ed.), Virtual Router Redundancy Protocol (VRRP) Version 3, RFC 5798, IETF 2010. RFC
• OPC Foundation, OPC UA Part 4: Services, 6.6.2 Server Redundancy (transparent, cold, warm, hot, hot-and-mirrored). spec
• ClusterLabs, Pacemaker documentation (fencing/STONITH). docs