Traffic flow

Health monitoring tells you a link is at 80% utilization; flow monitoring tells you what is on it. Yagra collects the flow records your devices already know how to export and turns them into top talkers, conversations, port and protocol mixes, and AS-level views — alongside the metrics and alerting you already have, and entirely separate from them: leave flow off and nothing changes.

Overview

Yagra understands the common flow export protocols:

NetFlow v5 and NetFlow v9
IPFIX
sFlow v5

Point a router, firewall, or switch’s flow export at a poller and Yagra answers the questions flow data exists for: who talks to whom, which hosts and ports carry the traffic, what the protocol mix looks like, and — with AS enrichment — which autonomous systems your traffic fans out to.

Flow records are stored in a dedicated ClickHouse store, separate from the metrics TSDB and the metadata database. It is required for the feature: set YAGRA_CLICKHOUSE_URL to enable it. Without it, flow collection is disabled and the flow API answers a typed 503, so the WebUI can say “not configured” rather than showing empty charts.

The single-node compose stack already ships a ClickHouse service with the URL set, so flow works out of the box there; on a hand-rolled deployment the store is opt-in. Settings ▸ System Health lists the flow store alongside the other backing stores, so a ClickHouse outage is visible at a glance rather than surfacing as mysteriously empty flow views.

Enabling collection

The flow listeners live on the poller and are opt-in, like the other passive listeners — each exists only while its bind variable is set:

YAGRA_FLOW_BIND=0.0.0.0:2055     # NetFlow v5/v9 + IPFIX — set by default in compose
YAGRA_SFLOW_BIND=0.0.0.0:6343    # sFlow v5 — off by default

Listener	Variable	Port	Compose default
NetFlow v5/v9 / IPFIX	`YAGRA_FLOW_BIND`	`2055/udp`	on
sFlow v5	`YAGRA_SFLOW_BIND`	`6343/udp`	off (the host port mapping exists, but nothing listens until you set the bind)

Under compose the host ports are mapped by YAGRA_FLOW_PORT (default 2055) and YAGRA_SFLOW_PORT (default 6343). Unlike syslog and traps these are unprivileged ports already, so host and container ports match and no redirect rules are needed.

On the device side, configure the flow export — vendors variously call it NetFlow export, NetStream, IPFIX export, or an sFlow agent — to send to <poller-host>:2055 (or :6343 for sFlow). Expect the first data on the node’s Flow tab a minute or two after the first export arrives: records are folded into 60-second buckets, so the view fills in bucket by bucket.

sFlow is a sampling protocol: each sample represents 1-in-N packets, so Yagra scales byte and packet counts by the sample’s sampling rate to estimate the real traffic. Treat sFlow numbers as good estimates, not exact counters — that is the nature of the protocol, not a Yagra limitation. Sampled NetFlow v5 (a sampling interval in the export header) is scaled the same way.

Because flow, like syslog and traps, is received by the poller, it works at remote sites with no extra plumbing — see Remote-site collection below.

How flows are stored

Raw flow export from a busy device is enormous, and most of it is long-tail noise. Yagra aggregates at the edge, on the poller, before anything crosses the bus:

Records are folded into 60-second buckets (YAGRA_FLOW_BUCKET_SECS).
Within each bucket, the poller keeps the top 500 flows per exporter by bytes (YAGRA_FLOW_TOP_N). This is the real cardinality control: it caps what a device can cost, no matter how much it exports, while keeping the heavy flows that answer “what is on this link”.
The aggregated batches ride the bus to core, which writes them to ClickHouse.

The trade is deliberate: the heavy flows that dominate a link are preserved exactly, and the long tail of tiny flows is what the top-N truncation folds away. For “what is saturating this link” and “who talks to whom”, that is the right data at a small fraction of the raw volume. Each stored record carries the conversation’s source and destination address and port, the protocol, the AS numbers, and the byte and packet counts for its bucket, keyed to the exporter that reported it.

Retention is applied in ClickHouse via TTL: 30 days by default, configurable from 1 to 3650 days with YAGRA_FLOW_RETENTION_DAYS. The flow store is deliberately loss-tolerant — it holds traffic analysis data, not billing records, and losing it never affects monitoring or alerting.

AS enrichment

Flow views can group and filter by autonomous system, which turns “traffic to 203.0.113.7” into “traffic to AS15169”. AS numbers come from two places, in order:

The exporter itself. BGP-speaking devices export source and destination AS in their flow records; those values always win.
An offline IP→ASN dataset. Most non-BGP devices export AS 0. Point YAGRA_IPASN_DB at an iptoasn.com TSV dataset and Yagra fills in the missing AS numbers at write time. Unset, enrichment is off and only exporter-provided AS values appear.

YAGRA_IPASN_DB=/path/to/ip2asn-combined.tsv
YAGRA_IPASN_RELOAD_SECS=21600   # re-read the file every 6 h; 0 = load once at startup

The compose files keep the dataset fresh without giving the whole stack internet access: an ipasn-updater sidecar fetches it on a schedule (weekly by default, YAGRA_IPASN_REFRESH_SECS) into a shared volume — it is the only container that needs egress for this feature — and core hot-reloads the file periodically (YAGRA_IPASN_RELOAD_SECS) with no restart.

AS names are resolved at read time, not baked into stored records — so when the dataset updates a name, your history shows the current name too.

Exploring flows

Every node gets a Flow tab on its detail page:

Top talkers, top ports, top protocols — ranked cards for the node’s traffic.
Top AS — the autonomous systems behind the traffic, with drill-down.
Conversations — a table of who-talks-to-whom, plus a Sankey diagram of the same conversations, with source and destination AS shown alongside the addresses.

The views are interactive: click any talker, port, protocol, or AS to filter the whole tab down to it, and combine protocol, port, peer, and AS filters to chase a specific question.

Beyond the per-node tab:

The dashboard has a fleet-wide traffic-flow widget section — top talkers, top AS, top ports, protocol mix, a conversation Sankey, and a traffic trend — reading across every exporter, not one node at a time.
The Troubleshoot screen includes flow-backed analyses (traffic anomaly, talker shift, new destination, scan detection) and cross-signal ones that read metrics, events, and flow together.
AI clients get the same data through the MCP tools top_flows and flow_fanout.

Remote-site collection

A branch site does not need a tunnel to the central ClickHouse. Point the site’s devices at the site’s own poller: the poller receives the exports locally, aggregates them, and the flow batches ride its single outbound TLS bus connection home — the same connection that already carries polling jobs and results. No inbound ports at the site, nothing new through the central firewall.

Flow does add real WAN traffic for a remote poller, because pollers also carry the original datagrams verbatim (that is what makes byte-exact forwarding possible). Budget roughly 0.4–1.0 Mbit/s per 1,000 flows/s, depending on how densely the exporter packs its datagrams — the full bandwidth budget discussion is in Ports & firewall.

Rate limits and tuning

Flow intake has its own token-bucket limits, separate from the syslog/trap budget, counted in datagrams (not records — one NetFlow datagram can carry dozens of records):

Limit	Default	Variable
Per source (exporter)	`1000` datagrams/s	`YAGRA_FLOW_RATE_PER_SOURCE`
Global (all exporters)	`20000` datagrams/s	`YAGRA_FLOW_RATE_GLOBAL`

Both allow bursts of up to 2× the sustained rate. Datagrams over the limit are dropped at the listener — the edge protects the bus and the store, so one runaway exporter cannot crowd out the rest, and losing samples from the loudest source is the least-bad failure mode for data that is statistical to begin with.

The full set of flow knobs:

Variable	Default	Controls
`YAGRA_CLICKHOUSE_URL`	unset (flow disabled)	The ClickHouse flow store — required
`YAGRA_FLOW_BIND`	set in compose (`0.0.0.0:2055`)	NetFlow/IPFIX listener
`YAGRA_SFLOW_BIND`	unset (off)	sFlow listener
`YAGRA_FLOW_BUCKET_SECS`	`60`	Aggregation bucket width (seconds)
`YAGRA_FLOW_TOP_N`	`500`	Flows kept per exporter per bucket
`YAGRA_FLOW_RATE_PER_SOURCE`	`1000`	Per-exporter datagram rate cap
`YAGRA_FLOW_RATE_GLOBAL`	`20000`	Global datagram rate cap
`YAGRA_FLOW_RETENTION_DAYS`	`30`	ClickHouse retention (1–3650)
`YAGRA_IPASN_DB`	unset (enrichment off)	Path to the IP→ASN dataset
`YAGRA_IPASN_RELOAD_SECS`	`0` (load once)	Hot-reload period for the dataset

A very high-volume edge can also raise the shared listener knobs — YAGRA_LISTENER_WORKERS (parallel reader sockets per listener, defaulting to the host’s parallelism capped at 4) and YAGRA_LISTENER_RCVBUF_BYTES (per-socket receive buffer, default 4 MiB). These apply to all the passive listeners, not just flow.

Details and the compose-side port variables are in the configuration reference.