Alarm management

Deep dive

How it works

Parts:

• rationalization record — per alarm: consequence, operator action, response time, priority
• conditioner — limit, deadband, on/off-delay; runs in the PLC or alarm server
• state machine — Normal, ActiveUnacked, ActiveAcked, RtnUnacked per alarm
• annunciator and alarm list — priority-based sound, colour and ordering
• suppression and shelving — design-time and time-limited operator hiding, logged
• flood monitor and KPI report — rates per operator, top-10, chattering, stale alarms

Operations: the steps below, run once per sample (2-3), per operator action (3, 5) and per review period (6).

1. Rationalize: every alarm gets a consequence, an operator action, a response time and a priority. If it has no action, it is not an alarm.
2. Condition at the source (in the PLC or the alarm server): the entry condition is v >= limit; the exit condition is v < limit - deadband (hysteresis). A change must hold for the on-delay (raise) or off-delay (clear) before it counts; a flip back restarts the timer.
3. State machine: Normal -> ActiveUnacked -> (ack) ActiveAcked -> (condition clears) RtnUnacked -> (ack) Normal. An alarm that clears before being acknowledged stays visible as RTN-unacknowledged so the operator sees that it happened.
4. Annunciate by priority: sound and colour only for what needs attention; the commonly cited target mix is roughly 80 % low, 15 % medium, 5 % high priority.
5. Reduce floods: suppress consequential alarms by design (state-based, e.g. a pump that is stopped cannot have low flow), group repeats, and allow time-limited operator shelving with a reason.
6. Monitor: alarms per operator per 10 min, peak rate, top-10 most frequent alarms, chattering alarms, stale alarms (active for a long time), and flood duration. Review monthly.

v: ──80──────/\/\/\────────────────────  chatter: no alarm (deadband 2, on-delay 3 s)
              ______________                 
v: ──80─────/              \____ 78 ───      raise after on-delay, clear below 78 + off-delay
state: Normal  ActiveUnacked  ActiveAcked  RtnUnacked  Normal

Code

A high-limit alarm with on-delay, off-delay, deadband, acknowledge and shelving, plus a sliding-window flood monitor (std only; compiled and asserted):

use std::collections::VecDeque;

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum State { Normal, ActiveUnacked, ActiveAcked, RtnUnacked }

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Event { Raised, ReturnedToNormal }

/// High-limit alarm: on-delay, off-delay, deadband, ack, shelving. Time in ms.
pub struct Alarm {
    pub limit: f64,
    pub deadband: f64,     // clears only below limit - deadband
    pub on_delay: u64,     // condition must hold this long to raise
    pub off_delay: u64,    // and be clear this long to return to normal
    pub state: State,
    since: Option<u64>,    // start of the current uninterrupted raise/clear candidate
    shelved_until: u64,
}

impl Alarm {
    pub fn new(limit: f64, deadband: f64, on_delay: u64, off_delay: u64) -> Self {
        Alarm { limit, deadband, on_delay, off_delay, state: State::Normal, since: None, shelved_until: 0 }
    }
    pub fn shelve(&mut self, now: u64, dur: u64) { self.shelved_until = now + dur; }
    pub fn shelved(&self, now: u64) -> bool { now < self.shelved_until }

    /// Feed one sample. Returns an event only if the operator should be told.
    pub fn update(&mut self, v: f64, now: u64) -> Option<Event> {
        let active = matches!(self.state, State::ActiveUnacked | State::ActiveAcked);
        // Hysteresis: the entry and exit conditions differ.
        let cond = if active { v >= self.limit - self.deadband } else { v >= self.limit };
        if cond == active { self.since = None; return None; } // no change pending
        let t0 = *self.since.get_or_insert(now);
        let need = if active { self.off_delay } else { self.on_delay };
        if now - t0 < need { return None; }
        self.since = None;
        if active {
            self.state = State::RtnUnacked;
            Some(Event::ReturnedToNormal)
        } else {
            self.state = State::ActiveUnacked;
            // Shelved alarms still change state (and are logged) but are not annunciated.
            (!self.shelved(now)).then_some(Event::Raised)
        }
    }
    pub fn ack(&mut self) {
        self.state = match self.state {
            State::ActiveUnacked => State::ActiveAcked,
            State::RtnUnacked => State::Normal,
            s => s,
        };
    }
}

/// Alarm-flood detector: flood while more than `limit` alarms arrived in the last `window` ms.
pub struct FloodMonitor { window: u64, limit: usize, times: VecDeque<u64> }

impl FloodMonitor {
    pub fn new(window: u64, limit: usize) -> Self { FloodMonitor { window, limit, times: VecDeque::new() } }
    pub fn record(&mut self, now: u64) -> bool {
        self.times.push_back(now);
        while self.times.front().map_or(false, |&t| now - t >= self.window) { self.times.pop_front(); }
        self.times.len() > self.limit
    }
}

Usage: call update(value, now_ms) once per scan (see PLC scan cycle); a returned event goes to the alarm list, journal and notification path; ack() comes from the operator action.

Complexity and constants

• Per alarm: a few words of state (state, timer start, shelve time); one comparison per sample. 100,000 alarm points at 1 Hz is order of 10^5 comparisons per second, trivial on one core.
• Cost drivers are not CPU: they are the events you create. Each event costs a journal record (order of 100-200 bytes), a network message and, above all, operator attention.
• Detection delay is bounded: on_delay plus the scan period. Pick it from the process time constant, not from the scan period.
• Flood window: a ring of timestamps, 8 bytes each; at 10 alarms per window it is under 100 bytes.

Efficiency

• Cost: per alarm point one comparison per sample and a few words of state; 100,000 points at 1 Hz is order of 10^5 comparisons per second. A raised event costs a journal record (order of 100-200 bytes), a network message and operator attention. Bound: human attention, then message volume.
• Levers:
  • rationalize alarms away — every deleted alarm removes all its events — alarms with no operator action
  • deadband and on/off-delay in the source device — repeats never leave the PLC; cost is a detection delay of at most on_delay plus one scan — noisy analog points
  • state-based suppression — consequential alarms never fire — trip sequences and stopped equipment
  • grouping and dedupe in the alarm server (event deduplication) — one notification per group — repeat floods
  • report by exception, evaluate in the controller — no polling round trip per point per period — large point counts
• Naive → standard → state of the art:
  • raw limit check, every crossing annunciated · one event per crossing, hundreds per trip
  • deadband, delays, ack state machine, priorities · chatter removed, bounded delay
  • rationalized, state-based, shelving with review, KPI-monitored · alarms per operator per 10 min tracked against the guideline above

Implementation notes

• Use wrap-safe time (monotonic ms, now - t0) and take timestamps at the source, not at the server, so the sequence of events survives network delay; send the PLC timestamp over OPC UA (the Alarms and Conditions model carries state, severity, acknowledge and shelving).
• Persist active/unacknowledged state across a restart of the alarm server and replicate it to the hot standby, or the operator loses what was unacknowledged.
• Journal every state change, acknowledge, shelve, suppression and setpoint change with user and time; the alarm log is an audit trail.
• Choose the deadband from sensor noise (peak-to-peak) and the engineering consequence; measure it per point.
• Bad-quality input (sensor fault, comm loss) is a separate alarm and must not trigger the process alarm; handle it before the limit check, and keep the supervision itself alive (see watchdog timer).
• Shelving: fixed maximum duration, automatic unshelve, visible shelved list, reason required.

Pitfalls

• Alarm priority used as a proxy for importance to engineering, not to the operator's response time.
• Limits copied from design data rather than rationalized; alarms equal to the trip setpoint leave no time to react.
• Chattering alarms fixed by raising the limit; the correct fix is deadband and delay.
• Acknowledge-all buttons; they train acknowledging without reading.
• Shelving as the permanent solution, with no expiry and no review.
• Silent suppression after changing a mode, then forgetting to re-enable.
• Counting alarms per plant rather than per operator console; the guideline is per operator.

Variants and alternatives

In production

Process plants and power stations run ISA-18.2 / IEC 62682 or EEMUA 191 alarm systems on DCS and SCADA; control-system toolkits such as EPICS and Phoebus implement alarm severity, hysteresis and acknowledge; IT monitoring uses Prometheus Alertmanager for grouping, inhibition and silencing. Airplane flight decks follow 14 CFR 25.1322 flightcrew alerting; medical electrical equipment follows IEC 60601-1-8.

Implementations

Sources

• ANSI/ISA-18.2-2016, Management of Alarm Systems for the Process Industries (international equivalent: IEC 62682). IEC 62682:2014
• EEMUA Publication 191, Alarm Systems: A Guide to Design, Management and Procurement, 3rd ed., 2013.
• OPC Foundation, OPC UA Part 9: Alarms and Conditions specification. spec
• Prometheus, Alertmanager documentation. docs
• US eCFR, 14 CFR 25.1322 Flightcrew alerting. text
• AAMI/IEC, 60601-1-8: alarm systems in medical electrical equipment, Amendment 2, 2021. doi