Architecture
Yagra is built to run co-located in a few containers for a single-node deployment and to scale out to distributed pollers and highly available stores by configuration, not rewrite. The pieces are decoupled from the start.
Components
Section titled “Components”| Component | Role | Runs in |
|---|---|---|
| Core | Orchestration, scheduling, and the northbound REST API | core |
| Poller | ICMP / SNMP / API polling and passive intake — stateless and horizontally scalable | poller |
| WebUI | Dashboards and visualization (React + TypeScript) | browser |
| Bus | Job distribution and poller fan-out over NATS | core + poller |
| Transport | The ICMP / SNMP / HTTP abstraction all device I/O goes through | poller |
| Topology | Dependency graph powering suppression and the network map | core |
| Discovery | Device discovery and classification | core + poller |
| Alert | State machine, hysteresis, and dependency suppression | core |
| Ingest | Syslog and SNMP-trap parsing, flow decoding, and edge rate limiting | poller |
| Forward | Filtered tee of received events and flows to external collectors | core |
| Secrets | Envelope-encrypted device credentials — only core holds the key | core |
| Telemetry | Structured logs, Prometheus metrics, OpenTelemetry traces | core + poller |
Only the first three are things you deploy. Two Rust processes carry everything else: core (orchestration-side) and the poller (device-side), plus the static WebUI served by nginx. The rest of the table is libraries compiled into one or both of them — there is no ingest container, no alert service, no separate discovery daemon. The “runs in” column says which process each one ends up inside.
One pair of names is worth keeping apart: Bus is the client crate compiled into core and into every poller, while NATS is the broker they talk through — not a component but a backing service, listed with the stores below.
The core ⇄ poller boundary
Section titled “The core ⇄ poller boundary”Core and pollers communicate only through the bus — never by direct calls. This is what makes pollers stateless, horizontally scalable, and deployable at remote sites across NAT and firewalls: a remote poller dials out to the central bus rather than accepting inbound connections, so a site needs one outbound rule and no inbound holes.
A poller holds no durable state and touches no database. Everything it needs — job specs, device credentials, working-set assignments — arrives over the bus, and everything it produces — results, events, flow batches, heartbeats — leaves the same way. Because job messages can carry device credentials, exposing the bus to remote sites requires the bundled TLS + auth configuration (see security).
Store separation
Section titled “Store separation”Each kind of data lives in the store built for it. Three backing services are required; three are optional and switch their feature on when configured — the same framing as the installation guide:
| Backing service | Holds | Required? |
|---|---|---|
| PostgreSQL | Metadata: nodes, configuration, thresholds, users, alert history | Required |
| NATS | The core ⇄ poller bus: jobs, working sets, results, events | Required |
| VictoriaMetrics | Time-series metrics | Required |
| Redis | Ephemeral poller liveness/assignment mirror — rebuildable, losing it is safe | Optional |
| VictoriaLogs | Passive-event log store — full-text syslog/trap search | Optional |
| ClickHouse | Traffic-flow records | Optional |
The rule behind the table: high-cardinality time-series never goes into PostgreSQL, and durable configuration never goes into Redis. Without VictoriaLogs, passive events stay entirely in PostgreSQL; without ClickHouse, flow monitoring is off.
Pollers store raw SNMP counters; rates and utilization are derived at query time (which handles counter wrap and resets correctly), so pollers hold no previous-value state.
Data paths
Section titled “Data paths”The poll lifecycle
Section titled “The poll lifecycle”Core schedules every check — with jitter, so thousands of nodes don’t probe on the same tick — and publishes work to the bus. The assigned poller runs the probe through the transport abstraction (ICMP, SNMP, or HTTP), and publishes the result back. Core evaluates state and thresholds, drives the alerting pipeline, and writes the raw metric samples to VictoriaMetrics. Nothing rate-related is computed at collection time: rates come out of the time-series store at query and evaluation time.
Passive events
Section titled “Passive events”A device sends syslog or an SNMP trap to a poller’s UDP listener. The poller parses it, applies per-source and global rate limits at the edge, and forwards the event — along with a copy of the original datagram bytes — over the bus. Core correlates it to a node by source IP, matches it against event rules to raise or clear alerts, and persists it to PostgreSQL (and to VictoriaLogs, when configured, for full-text search). Inbound webhooks skip the poller: they arrive directly on core’s API with a per-source bearer token. Details: passive events.
Traffic flow
Section titled “Traffic flow”A flow exporter sends NetFlow, IPFIX, or sFlow datagrams to a poller’s flow listener. The poller decodes and aggregates at the edge — fixed time buckets, top flows by bytes per exporter — and streams the aggregate to core, alongside the verbatim datagrams so that forwarding can relay exactly what the device sent. Core enriches flows with AS numbers (offline IP→ASN dataset) and writes them to ClickHouse. Details: traffic flow.
Forwarding egress
Section titled “Forwarding egress”Forwarding relays received syslog, traps, and flow exports onward to external collectors — and it egresses from core, not the pollers. Pollers carry the original received bytes to core over the bus; the active core applies the per-destination filters and does all relaying and BigQuery streaming. One egress point to firewall, regardless of how many sites feed it.
Scaling model
Section titled “Scaling model”Poller pools. Every node carries a pool attribute, and each pool’s nodes are spread across
its live pollers by consistent hashing — add a poller to a pool and core rebalances; lose one and
its nodes fail over to the survivors. Pools map naturally to sites: a Tokyo pool’s pollers monitor
Tokyo’s devices locally. See distributed polling.
Working sets. Core pushes each poller its assigned node set over the bus — a snapshot plus incremental deltas — and the poller schedules its own probes locally. Polling capacity scales horizontally without core micromanaging every check, and a briefly poller-less pool falls back to per-job publish so no node goes dark.
High availability. Multiple cores can run against the same stores with automatic leader
election: one leader does the work, standbys take over within seconds, and /readyz tells the
load balancer which is which. See high availability.
Upgrades
Section titled “Upgrades”Version upgrades are designed to be low-effort and non-destructive: expand-contract database migrations that run automatically on core startup, version-tolerant bus messages (a new core works with the previous release’s pollers during a rolling upgrade — so upgrade core first, then pollers in any order), and persistent stores preserved across upgrades with a pre-upgrade backup step. The procedure lives in the installation guide.