Alerting

Alert quality is a first-class concern in Yagra. The goal is not to send alerts — it is to send the right alerts: one page per real incident, no flood when a site drops, no 3 a.m. noise from a metric grazing its threshold for a single sample. Hysteresis, flapping detection, dependency suppression, and deduplication are always in the delivery path — they are not optional extras you switch on later, and there is no way to bypass them to “just send the alert”.

Escalation and on-call scheduling are deliberately delegated to external tools (PagerDuty, Jira Service Management): Yagra emits a clean fire/resolve lifecycle signal and holds no escalation scheduler of its own.

The alert lifecycle

Three things can raise an alert:

a liveness failure — a device stops answering its ICMP probe (or a URL/DNS monitor stops succeeding);
a threshold breach — a gauge metric crosses a rule’s warning or critical bound (see Monitoring for the threshold model);
a matching event rule — a passive event (syslog, trap, webhook) matches an operator-defined rule (see Event-raised alerts below).

Whatever the trigger, every check on every node carries a committed state:

State	Meaning
`ok`	The check is healthy
`warning`	A threshold rule’s warning bound is breached
`critical`	A threshold rule’s critical bound is breached
`unreachable`	Liveness has failed — the node does not answer
`unknown`	Yagra has no current opinion (no recent samples)
`maintenance`	The node is inside a maintenance window

Only the problem states — warning, critical, unreachable — carry a severity and produce an alert. ok, unknown, and maintenance never do: a node Yagra can’t see yet, or one under planned work, is not an incident. (You will see unknown briefly on newly added nodes and right after a core restart, before the alert engine has formed an opinion on every node — it resolves itself as samples arrive.)

An alert is raised when a check commits a problem state (after the hysteresis below), and clears when the check recovers through the same path. Repeated fires of the same problem are deduplicated and grouped into one incident rather than delivered as a stream of duplicates. Every fire and clear is recorded in an append-only alert history, with the measurement that caused it captured at fire time.

Dwell-time hysteresis

A raw sample crossing a threshold does not flip the committed state. The breach must persist for a configured number of consecutive samples (the dwell) before the state commits and an alert fires — and any sample that returns to the committed state resets the candidate, so an interrupted streak starts the count over.

This is what keeps a metric that grazes its bound once, or a single dropped ping, from paging anyone. A dwell of three, for instance, means three breaching samples in a row at the node’s poll interval — one recovery sample in between and the count starts again.

Counting in samples rather than wall-clock time also has an architectural consequence: alert evaluation always runs against the live sample stream, and history that is backfilled later (for example, after a remote poller reconnects and replays its buffer) is never re-evaluated into alerts — replaying a backlog would fabricate transitions for incidents that already resolved. Metrics fill in; alerts resume from now.

Flapping detection

A link that goes up and down every few minutes generates a technically-correct alert each time — and each one is noise. Yagra counts committed state transitions per check inside a sliding time window; a check exceeding the transition threshold within the window is flagged as flapping on its transitions.

Flapping checks surface on the Flapping watchlist dashboard widget, which names the node (the id is on hover) and shows what the flapping check measures — so the unstable link gets fixed as one problem instead of being re-triaged on every bounce.

For deeper digging, the Troubleshoot catalog includes a flap analysis that scans a node, a group, or the whole fleet for reachability and link-state churn — and a companion that finds event rules repeatedly firing and clearing on the same node.

Dependency suppression

When a site router dies, everything behind it goes dark — and without suppression, every one of those nodes pages you. Yagra drives suppression from the dependency graph: each node can name its upstream, and the graph is visualized on the Network map.

When a node goes unreachable and its upstream chain is also down, the alert is attributed to the highest down ancestor — the root cause — and rolled into that parent’s incident. The child’s alert still fires and is still visible in the UI and the history; only the duplicate notification is suppressed. The topology views and the dependency / root-cause dashboard widget show the attribution, so one incident reads as one incident with its blast radius attached.

Suppression is re-evaluated event-driven, whenever a parent’s state changes — in both directions:

A parent that goes down after its children were already alerting re-groups those existing alerts under its incident (their standalone pages are closed).
A parent that recovers while a child is still down re-pages that child on its own — it is no longer explained by the parent.

Two deliberate limits keep suppression honest: only liveness alerts are ever suppressed — threshold alerts are not, since a reachable node with a real threshold breach deserves its page — and alerts raised by passive events skip suppression too, because a device that just emitted an event is evidently reachable.

Dependency edges are editable from the WebUI: the node-detail header’s Dependency… action sets or clears a node’s upstream, and Topology ▸ Dependency view lists every node with its upstream, live status, and current root-cause attribution, editable inline. Self-dependencies and cycles are rejected.

Letting Yagra derive the graph

Typing an upstream per node does not scale, and a graph nobody maintains is a graph that suppresses the wrong things. Yagra can derive the dependency graph from the network map instead — direction comes from each node’s distance to a poller, so whatever sits one hop closer is its upstream.

Topology ▸ Dependencies has a three-position mode switch, and nothing moves between the positions on its own — an upgrade lands on the mode you were already on:

The hand-authored graph — the default, and where every existing deployment stays.
Comparing — changes nothing about alerting. It shows, node by node, where the derived graph and the one you maintain disagree, plus the two numbers that decide whether the switch is safe: how many active alerts the derived graph would newly suppress (the risky direction — each one is an alert that would stop being raised) and how many it would stop suppressing.
The derived graph — hands suppression over.

Two things the derived graph can express that a hand-typed parent_id cannot. A node gets every neighbour one hop closer to a poller, so a server reached through a redundant pair of routers has both as parents and keeps alerting while either is alive — which is what the “suppressed only when every parent is down” rule was always written for. And a single box can be taken out of derived suppression entirely with Never suppress, so its alert always stands on its own whatever the graph says. That switch only ever removes suppression, so it cannot cause an outage to go unreported.

Where the derivation gets a link wrong, record the decision rather than working around it: a link can be pinned into existence, hidden, or have its direction declared, and those always beat what was derived on every recomputation.

Maintenance windows and mutes

Planned work should never page anyone:

Maintenance windows cover scheduled work. A window is opened against a node; while it is open the node sits in the maintenance state, which never produces an alert. Opening and closing windows is an Operator-and-up action (the manage-maintenance permission).
Mutes are the ad-hoc form — silence a known-noisy alert without scheduling anything. Muting, like acknowledging, is part of the Operator role’s respond-to-incidents permission.

Both are first-class, audited objects with their own settings pages — not a notification filter bolted on at the end.

Routing and notification channels

Once an alert survives hysteresis, suppression, and maintenance checks, notification routing decides where it goes. Yagra delivers to:

Channel	Notes
Webhook	Generic HTTP webhook, with retry
Email (SMTP)	With retry; relay, sender, and credentials configurable
PagerDuty	Events API v2
Jira Service Management	Alerts API

Every channel receives the full lifecycle — fire and resolve both — so the external tool’s incident closes when Yagra’s alert clears, instead of leaving stale incidents to be tidied by hand. Acknowledgement performed in an external tool is reflected back into Yagra read-only. Routing rules choose which channels are notified, managed on the notification-routing settings page.

Delivery runs off the result-ingest path: a slow or unresponsive notification endpoint can delay its own notifications (they retry), but it can never stall metric ingestion or alert evaluation behind it.

Outbound webhooks are SSRF-guarded the same way monitoring probes are: loopback, link-local, and cloud-metadata targets are refused.

Notification templates

Each channel can override the subject and body it sends, written as a Jinja2 template over a fixed set of alert variables — node name, address, group, profile, severity, metric, threshold, the observed value, and whether this is a fire or a resolve. Conditionals work, so wording can differ by severity or by lifecycle event:

{% if event == 'resolve' %}Recovered{% else %}{{ severity | upper }}{% endif %}: {{ node_name }} ({{ group }})

A channel with no template sends Yagra’s built-in wording, unchanged. Two properties are worth knowing before you write one:

A broken template never costs you a notification. If it cannot be rendered when an alert fires — a bad filter, an oversized output, or a body that stopped being valid JSON on a channel that sends JSON — that field falls back to the built-in wording and the notification still goes out. The fallback is a plain string substitution, not a second trip through the engine that just failed.
Templates are sandboxed. A template can reach nothing but the variables it is handed: no files, no includes, no other templates. Execution is bounded, so a runaway loop cannot stall the notification worker.

The editor lives at Alerts ▸ Notification routing ▸ a channel ▸ Edit notification template, with a variable palette and a live preview rendered against a sample alert.

For a fixed deployment-wide route without touching the UI, two environment-level channels exist: YAGRA_WEBHOOK_URL defines an always-on default webhook that fires for every alert alongside the configured channels, and the YAGRA_SMTP_* variables define a default email route — see the configuration reference.

Event-raised alerts

Alerts don’t only come from polling. Passive events — syslog messages, SNMP traps, and inbound webhooks — are matched against operator-defined event rules (substring or regex), and a matching rule can raise an alert. Incoming SNMP traps have their trap OID resolved to a human-readable name, and a set of built-in trap event rules ships out of the box, so common traps arrive as named, alertable events without configuration. See Passive events for the ingestion side.

Event-raised alerts differ from poll-driven ones in two ways:

They clear on a TTL. A one-shot event has no “recovery sample” to observe, so each rule carries an auto-clear TTL; the alert closes when it expires (there is no manual close).
They skip dependency suppression. A device that just emitted an event is reachable, so suppressing its alert under a “down” parent would be wrong.

A rule scoped to a single event stream stays scoped to it: a rule whose source kind this core version does not recognise is left out of the matching engine and logged, rather than silently widening to every stream.

When monitoring itself goes dark

Every alert above depends on poll results arriving. That makes the loss of a poller the one failure the alert engine cannot reason about: when a poller pool loses its last live poller, its jobs are published to a subject nothing is subscribed to and discarded, and the nodes drift to unknown rather than down. An entire site stops being monitored while every dashboard stays calm.

Yagra watches for this directly. A pool that still holds nodes but has no live poller raises a critical alert of its own, delivered over the same notification channels and routing rules as any other, and closed automatically when a poller returns.

It is a full alert, not just a notification — it appears on Active alerts and in alert history, streams live, and renders through your notification templates.
It waits five minutes by default. A poller announces its own departure, so an ordinary rolling restart raises the condition instantly; the debounce is what stops that paging anyone. Tune or disable it with YAGRA_POOL_COVERAGE_ALERT_AFTER_SECS (configuration).
Its subject is a pool, not a node. This is the one alert that belongs to no device, so it never rolls into a node’s displayed state and it cannot be muted — a mute names a node. API clients should branch on subject_kind before reading an alert’s node field as a node id.
Meraki-managed nodes are excluded: core’s org collector polls them, so they do not depend on a pool.

Two gauges are exported whether or not the alert is enabled — yagra_pools_without_live_poller (unlabelled, the one to alert on or drive a scale-up from) and yagra_pool_nodes_without_live_poller{pool}, which reports 0 for a healthy pool rather than disappearing.

Working with alerts

Active alerts is the triage view: every currently-firing alert, newest first, with severity, the node’s name (id on hover), what the check measured and how it breached — for example, icmp_rtt_ms above 100 (was 450) — and its age. The list is virtualized, so it stays smooth even with thousands of simultaneous alerts during a major outage. The notification bell in the top bar shows the active count and opens this view. Each node’s detail page also shows the alerts currently attached to that node.
Acknowledging an alert is an operator action — the respond-to-incidents permission, Operator and up — and marks it as being handled; the acknowledgement can also be cleared. Roles and permissions are covered in Users and SSO.
Alert history is the append-only record of every fire and clear. Each entry keeps the metric and condition captured at fire time in its “What” column, so history stays truthful even after the threshold rule that raised it is edited or deleted. Active alerts and history render the measurement through the same formatter, so the two screens cannot disagree.

Operator actions — acknowledging, muting, opening a maintenance window — are recorded in the audit log like every other state-changing action, so who silenced what, and when, is always answerable. The same actions are available to AI and automation clients through Yagra’s MCP tool surface, under the same permission checks and the same audit trail.