Distributed pollers

Yagra scales polling horizontally across pools of stateless pollers, keeps probes close to the devices they measure, and keeps monitoring alive through network partitions. This page explains how that behaves — assignment, failover, bus security, and the store-and-forward buffer. For the step-by-step deployment runbook, see the installation guide, section D.

None of this is a separate “distributed mode”. A single-node install already runs one poller in the pool "default" with exactly the machinery described here; going distributed means adding pollers and pools, not changing architecture. The single-node and distributed deployments run the same images.

Pools and assignment

Every node carries a pool attribute (default: "default"). A poller identifies itself with a stable id (YAGRA_POLLER_ID) and the pool it serves (YAGRA_POLLER_POOL), and sends heartbeats over the bus. Core’s coordinator tracks those heartbeats and assigns each pool’s nodes across its live pollers using consistent hashing — so adding or losing a poller reshuffles the minimum number of nodes, not the whole pool.

Each poller receives its assignment as a working set pushed over the bus: a snapshot first, then incremental deltas as nodes are added, moved, or reassigned. The poller schedules its own probes locally from that set — core does not micromanage individual checks — and a poller that reconnects after a restart gets a full resync, so there are no polling gaps across restarts.

Failure handling follows from the same machinery:

• A poller drops out — its heartbeats stop, and the coordinator fails its nodes over to the surviving pollers in the same pool automatically.
• A poller joins — the pool rebalances across the new membership, again moving only the minimum share.
• A pool has zero live pollers — core falls back to publishing that pool’s jobs individually over the bus, the path that predates working sets. This is the rolling-upgrade safety net: a pool whose pollers are mid-restart, or still running the previous release, keeps its nodes monitored instead of going dark.

The liveness and assignment state lives in the (optional) Redis mirror and is rebuildable — losing it degrades nothing durable. Scaling a pool is simply running more pollers with the same YAGRA_POLLER_POOL and distinct YAGRA_POLLER_IDs.

Assigning a pool, and seeing who polls what

A node’s effective pool resolves in order: the node’s own pool, then the nearest ancestor folder that sets one, then "default". Both are editable from the WebUI — on a node, on a folder, and by right-clicking either in the inventory tree, which offers the pools that already exist plus Inherit and a custom name. Folders are the important case: the tree has no multi-select, so folder inheritance is how you move a site’s worth of nodes at once.

Each choice shows whether that pool has a live poller. This matters more than it looks: assigning nodes to a pool with no registered poller publishes their jobs to a bus subject nothing subscribes to, and those nodes quietly stop being monitored.

Node detail answers the reverse question. Pool shows the effective pool and where it came from, and Polled by names the poller actually responsible — one of assigned, pending (assignment not yet published), legacy fan-out (the zero-poller fallback above), Meraki (cloud-collected, so not ring-assigned), or unknown. That answer is read from the working set core actually published rather than recomputed from the hash ring, so it never invents an owner for a node the sweep dropped, and it cannot disagree with the poller’s own node list. Settings ▸ Pollers goes the other way, drilling into any poller’s node set inline.

Upgrades lean on the same properties. The bus is version-tolerant between adjacent releases, so a new core works with the previous release’s pollers: upgrade core first, then pollers — including remote sites, one at a time, in any order. Pollers are stateless, so replacing one is free, and a pool briefly without pollers rides the per-job fallback until its upgraded pollers re-register. The full procedure is in upgrades & backups.

Location affinity

Pools map naturally to sites. Because pollers are stateless and dial out to the central bus, a remote site needs one outbound rule and no inbound holes — a poller behind NAT or a firewall works unmodified. Run a pool per site (“tokyo”, “osaka”, a cloud region), assign that site’s nodes to it, and probes stay local: ICMP and SNMP traffic never crosses the WAN, and only results, events, and heartbeats stream back to core.

The intake edge follows the pool. A site poller is also where the site’s passive telemetry should land. Its optional UDP listeners — syslog, SNMP traps, NetFlow/IPFIX, sFlow — accept the local devices’ output, apply per-source and global rate limits at the edge, and carry the events home over the same authenticated bus. Flow datagrams are additionally aggregated at the edge (fixed time buckets, top flows by bytes per exporter) before streaming to core, which is what keeps a busy site’s flow volume affordable on the uplink. Passive events correlate to nodes by the datagram’s source IP, which is why the remote-poller composition runs on the host network — bridge NAT would rewrite the address.

Budget the site uplink for what does cross it: pollers carry the original received bytes alongside the parsed events (so forwarding can later relay exactly what a device sent), which for flow works out to roughly a megabit per second per 1,000 flows/s. The per-source and global intake rate limits in the configuration reference cap the bursts.

A typical multi-site fleet looks like:

Site	Pool	Pollers
Datacenter / HQ	default	co-located with core
Tokyo branch	tokyo	1–2 remote, on the branch LAN
Osaka branch	osaka	1–2 remote, on the branch LAN

Within each pool, redundancy is just headcount: run two pollers in a site’s pool and either one picks up the other’s nodes on failure. The zero-live-poller fallback described above is an upgrade-compatibility path, not site redundancy — a second poller at the site is.

One special case is Cisco Meraki monitoring, which polls the Meraki cloud API at the organization level rather than probing devices on-site. Those cloud-collection jobs are routed to the pool named by YAGRA_MERAKI_POOL (default default) — point it at whichever pool has internet egress.

Registering pollers

Pollers are registered from the WebUI at Settings ▸ Pollers ▸ “Register poller”. The dialog generates a ready-to-use .env for the remote host, supplying the three variables a remote poller requires — YAGRA_POLLER_ID (stable and unique), YAGRA_POLLER_POOL, and the tls:// bus URL — which drops straight into docker-compose.poller.yml (or the native binary’s environment). Pin the image with YAGRA_IMAGE_TAG there too. The poller appears on the page within a few seconds of starting, and core begins assigning that pool’s nodes to it.

The Settings ▸ Pollers page is also the fleet view of your polling layer. It shows, per poller:

• Liveness — current status and the last heartbeat received.
• Assignment — the pool it serves and its current working-set size, so you can see how a pool’s nodes are distributed.
• Version — the poller’s running version, useful mid-rollout.
• Monitoring gaps — the recent windows during which core lost contact with a poller (which poller, which pool, when, and for how long), so you can see at a glance when monitoring was blind and confirm the metrics were backfilled.

It also warns when a pool has nodes but no live poller. The full deployment walkthrough — compose file, host networking, and the privileged-port caveat for the intake listeners — is in the installation guide, section D.

Securing the bus

Job messages carry plaintext device credentials, because the poller needs them to probe. On a single host that is fine — the bus never leaves the internal Docker network. The moment the bus crosses a trust boundary to a remote site, it must be TLS-encrypted and authenticated first; never publish NATS :4222 plaintext. The five-step setup — server certificate, bus passwords, the NATS auth/TLS configuration, switching the co-located clients to tls://, and handing the public certificate to each site — is installation, section D, step 1. Each poller pins the server’s certificate via YAGRA_BUS_CA_FILE, so a remote site trusts exactly one bus endpoint rather than the system CA store.

The bundled NATS configuration gives the core user full access and the poller user least privilege: publish results, events, and heartbeats; subscribe only to jobs and working-set assignments. Be aware of its limit, though — the one poller account is shared across every poller. Any authenticated poller can read any pool’s assignments, so the shared account authenticates the fleet but is not a tenant boundary between sites. If that matters for your deployment, scope credentials per poller as described next.

Per-poller credential scoping

Optionally, core can act as the bus’s authentication service (NATS Auth Callout) and mint each poller a per-connection credential scoped to its own pool’s subjects. At connect time, core validates the connecting poller against the shared bootstrap secret (YAGRA_NATS_POLLER_PASSWORD), then issues it a credential that grants exactly the subjects for its own id and pool — its own job stream, its own assignment subject — and nothing else.

The effect: a compromised remote site can only ever see the jobs and device credentials for its own pool, instead of holding a shared account with fleet-wide reach. What each poller can read is decided centrally by core at connect time, not by editing the NATS server configuration per site.

Enabling it is opt-in, alongside the bus TLS + auth setup:

• Mount the NATS account nkey seed into core as a file and point YAGRA_NATS_CALLOUT_SEED_FILE at it — the secret is the mounted file, never an environment value.
• Set YAGRA_NATS_CALLOUT_ACCOUNT if your callout account is not the default ($G); it must match the account named in the NATS server’s callout configuration, which in turn needs the account’s public key as YAGRA_NATS_CALLOUT_ISSUER. The Compose files carry a commented block for both sides.

When the seed file is not configured, the callout responder simply never starts and NATS falls back to the static accounts above — off by default, and nothing changes until you turn it on. Details: security.

Store-and-forward

A remote poller cut off from core — a WAN outage, a firewall blip — keeps monitoring. It continues polling its devices locally and buffers the results instead of dropping them, in two tiers:

1. An in-memory ring (20,000 results by default). Buffering never blocks the poll loop.
2. An on-disk spill when the ring fills — segments written under /var/lib/yagra/buffer (the pollerbuf volume in Compose), so the buffer survives a poller restart mid-outage.

The buffer is bounded on every axis, and the overflow policy is drop-oldest everywhere, so it can never fill the poller’s disk:

Cap	Default	Variable
Master switch	on	YAGRA_STORE_FORWARD (off disables)
In-memory ring	20,000 results	YAGRA_STORE_FORWARD_MEM_MAX
On-disk spill, total	512 MB — oldest segment dropped past it	YAGRA_STORE_FORWARD_DISK_MAX_MB
Maximum age	24 h — older results are dropped at replay	YAGRA_STORE_FORWARD_MAX_AGE_SECS
Free-space floor	1 GB — spilling stops below this much free disk	YAGRA_STORE_FORWARD_DISK_FREE_FLOOR_MB
Spill segment size	16 MiB — the granularity of the disk cap	YAGRA_STORE_FORWARD_SEGMENT_MB
Spill directory	/var/lib/yagra/buffer	YAGRA_STORE_FORWARD_DIR

The buffer also degrades safely rather than failing: if the spill directory cannot be created or read, the poller logs a warning and continues with memory-only buffering — a storage problem never crashes polling.

On reconnect the poller bulk-replays the buffer over a dedicated backfill channel (permitted to the poller account on a secured bus, too), and core imports the metrics at their original timestamps — graphs and history fill in the outage window with no false spike. What replay carries is exactly the measurement record: metric samples and interface metadata, nothing else, and the spill holds no secrets while it waits on disk.

Alerts are never backfilled. Alert evaluation resumes from “now” after a reconnect. This is deliberate: dwell-time hysteresis is sample-count based, so replaying a backlog of old samples would fabricate state transitions and flood you with stale, already-resolved alerts. A recovered link therefore restores your history without re-raising incidents — and each healed outage window is recorded as a monitoring gap on Settings ▸ Pollers, so the blind period stays visible even though it never paged anyone.

Store-and-forward is on by default; set YAGRA_STORE_FORWARD=off to restore plain publish-live behavior (results are dropped if the bus is unreachable). All the caps are in the configuration reference.

Observing pollers

The polling layer is observable from three angles:

• Settings ▸ Pollers — per-poller health: status, pool, version, working-set size, last heartbeat, pool warnings, and the recent monitoring-gaps list described above.
• Dashboard — a working-set distribution widget shows how each pool’s nodes are spread across its pollers, which makes an unbalanced or degraded pool visible at a glance.
• Prometheus — every poller serves its own /metrics on :9100, alongside core’s metrics on the API port, so your existing monitoring can watch the monitors. See observability.

Together with the gaps list, this answers the operational questions in order: is every pool covered → who owns which nodes → when was I blind, and was it backfilled?