Troubleshoot & AI RCA

Monitoring tells you that something is wrong. The Troubleshoot section is for the next question — what is wrong, and why: a catalog of on-demand analyses that dig through the data Yagra has already collected, and an opt-in AI layer that turns an incident’s evidence into a written root-cause explanation.

On-demand analyses

A Troubleshoot analysis is a read-only job over Yagra’s own stores — the metrics time-series database, the passive-event log store, and the traffic-flow store. It never touches a device: no extra polling, no configuration reads, no traffic generated toward the network being diagnosed. That makes an analysis safe to run in the middle of an incident, which is exactly when you want it.

You launch an analysis from Troubleshoot ▸ Tools by picking a tool, a scope, and a depth:

• Scope — all (the whole inventory), group (the direct members of one group), or node (a single node). The scope picker searches the fleet server-side, so finding one node among tens of thousands is a typeahead, not a scroll.
• Depth — how many nodes the job will examine: quick caps the scan at 20 nodes, standard (the default) at 60, and exhaustive lifts the cap far enough to cover the whole fleet. Deeper scans take longer; quick exists so a first pass comes back in seconds.
• Tuning — most tools also take a time window, a baseline window to compare against, a sensitivity setting, and (for the metric tools) a metric-family filter to scan only reachability / interface metrics or only system metrics. “Compare this hour against the last day” and “only surface strong deviations” are knobs, not new tools.

Jobs run asynchronously. A running job can be watched live and cancelled, and lands in Troubleshoot ▸ Analysis runs with a state of running, done, failed, or cancelled. Each job returns a bounded set of findings — at most 60 per run — so a fleet-wide scan produces a readable shortlist, not a data dump.

The catalog

There are 15 analyses in four families, one family per store plus one that reads across them.

Metric analyses read the time-series store:

Analysis	What it looks for
anomaly	Baseline-relative deviation scoring — metrics behaving unlike their own history
correlation	Series that move together within a window — what changed at the same time
capacity	Time-to-exhaustion projection — which resources run out, and roughly when
flap	Reachability and link-state churn — interfaces and nodes that won’t stay up or down

Passive-event analyses read the syslog / trap event store:

Analysis	What it looks for
event_storm	A per-node event-volume spike against that node’s own baseline
event_flap	The same event rule firing and clearing repeatedly on one node
severity_shift	A node’s syslog severity mix skewing toward error / critical
rule_gap	High-volume unmatched events clustered by signature — missing rule coverage
auth_probe	Authentication-failure clustering by source — brute force, or a misconfigured NMS

Flow analyses read the traffic-flow store:

Analysis	What it looks for
traffic_anomaly	Node or interface flow volume outside its baseline
talker_shift	A talker or conversation newly dominant compared to a baseline window
new_destination	Traffic to a destination AS or port absent from the baseline window
flow_scan	One source contacting abnormally many distinct destinations or ports — scan / worm behavior

Cross-store analyses combine metrics, events, and flow:

Analysis	What it looks for
saturation	A single conversation dominating a busy node’s traffic — the link-hog
incident_correlate	A cross-signal incident timeline for one node: metric anomalies, events, and flow shifts in arrival order

incident_correlate is the one to reach for first during an incident: the order signals arrived in is often what points at the cause, and this tool assembles that order for you.

Each family reads its own store, so it can only find what that store holds: the passive-event analyses have material where syslog / trap ingestion is receiving events, and the flow analyses where flow export is configured. The metric analyses work on every deployment — the time-series store is not optional.

Running an analysis

Running an analysis requires the Operator role or higher — specifically the acknowledge-alerts permission, the same one that lets a user acknowledge or mute an alert. This holds on both surfaces: the REST API and the MCP tool surface alike, so a Viewer-scoped API token cannot launch analyses through an AI client either. Cancelling a run takes the same permission. Reading past runs and their findings needs only the Viewer role — an analysis changes no configuration, so its results are as open as the dashboards.

Because an exhaustive analysis over a large fleet is real work for the stores, the runner applies admission control:

• At most 4 analyses run concurrently (YAGRA_ANALYSIS_MAX_CONCURRENT).
• At most 30 new jobs may start per minute, over a sliding window (YAGRA_ANALYSIS_RATE_PER_MIN).
• When either limit is hit, the API answers HTTP 429 rather than piling on work — a client should back off and retry.

The defaults suit a single-core deployment watching thousands of nodes; both are environment variables if your hardware or habits differ.

The same limits govern every way in: the WebUI, the REST API, and the MCP tool surface share one runner, so an enthusiastic AI client cannot starve the human operator’s analyses — it hits the same 429. An analysis launched through MCP is also recorded in the audit log with the identity of the token that ran it.

Reading the results

Every one of the 15 analyses has its own report screen, built for its own kind of finding — a capacity forecast is not a list of log lines, and the reports don’t pretend otherwise. A few examples of what “tailored” means:

• incident_correlate renders an incident timeline — the signals plotted in arrival order.
• flow_scan draws a scan-shape scatter, so a worm’s fan-out pattern is visible, not inferred.
• saturation shows a share meter with capacity context: how much of the link one conversation is taking.

Findings link back to the node they concern, every report supports CSV export, and each report has a shareable deep link (?job=) — paste it in the incident channel and a colleague opens the same report, not a description of it. Deep links resolve the job directly, so a link to a run that has since aged out of the recent list still opens, and a link pasted under the wrong tool redirects to the report that can actually read it.

Past runs stay available under Troubleshoot ▸ Analysis runs, so the evidence a decision was based on can be revisited after the incident closes.

Searching what has been found

A run answers “what did this analysis find”. The question you usually start from is the other way round — has anything been found about this switch, or this site, lately? — and no single run can answer it.

Troubleshoot ▸ Saved findings searches findings across every run, newest first, filtered by node or site, by analysis, by severity, and by time window. Each row links back to the report of the run that produced it, so a promising finding is one click from its full context. The list is keyset-paged, so it stays usable however long the deployment has been collecting.

Running an analysis on a schedule

Some analyses are worth running whether or not anyone is watching — a nightly anomaly sweep, a weekly capacity forecast. Troubleshoot ▸ Scheduled runs one on a preset cadence: daily, weekly or monthly at a time of day (UTC), over the whole fleet, one site, or one node, with the same window and sensitivity settings a manual run takes.

Creating and editing a schedule takes the same permission as launching a run (Operator or higher). Two behaviors are worth knowing:

• A refused fire is deferred, not skipped. If admission control is full at the moment a schedule is due, the run is not launched — but the schedule stays due and the next minute’s tick retries it. A busy moment costs a schedule a minute, never a whole period. The schedules list reports that attempt as deferred, distinct from a failure.
• Traffic-flow analyses cannot be scheduled without a flow store. With no ClickHouse configured, a flow analysis returns a single “flow tier not enabled” note. That is a reasonable answer to a question asked once, and an unbroken column of empty runs when it is asked nightly — so those analyses are not offered in the schedule picker, and the API refuses them.

A schedule defaults to not notifying on completion, unlike a run you launch and wait for.

AI root-cause analysis

On top of the analyses sits an opt-in AI layer: an LLM writes a root-cause narrative for an incident, grounded in Yagra’s own data. From an active alert, an Operator can ask Yagra to “explain this incident”; Yagra assembles the evidence — the alert, the affected node’s facts, its position in the dependency graph, and the cross-signal timeline around the failure — sends it to the model you configured, and returns a structured answer: a summary, the likely root cause, affected dependents, suggested next steps, and a confidence estimate.

Two design choices are worth calling out:

• It explains the root cause, not the symptom you clicked. If the alert you start from is a child suppressed under a parent outage, the explanation targets the parent — the same root-cause roll-up the alert pipeline already computed (see Alerting).
• It is grounded in evidence Yagra already holds, and it can go and get more. The model starts from the incident context, and since v0.1.23 it can also look things up for itself through Yagra’s read-only MCP tools — pull the interface series, check whether the poller was even up, read what syslog said, inspect the threshold that fired. Previously it was handed a fixed set of facts and answered in one shot, so it could only reason about what had been decided in advance to include. It still has no network access of its own and no ability to touch devices.

AI RCA is off by default, and “off” means absent: with no provider configured there is no client, no stored credentials, and no egress — not a disabled feature flag in front of a live integration. The WebUI doesn’t show the “explain” affordance until a provider is enabled, and the API answers 503 for RCA requests on an unconfigured system.

Configuration lives in Settings ▸ AI (Admin only) and supports three providers, exactly one active at a time — deliberately no failover chain, so your incident data has exactly one destination you chose:

Provider	Where inference runs
Vertex AI (recommended)	Inside your own Google Cloud project and region — the request stays within a boundary you already govern
Google Gemini (direct API)	Google’s public API — leaves your operational boundary
Anthropic Claude (direct API)	Anthropic’s public API — leaves your operational boundary

Each adapter’s endpoint host is a constant in the code, not a setting — a misconfiguration cannot silently redirect incident data to an unexpected host. For Vertex, the credential is optional: a service-account key can be supplied, or omitted to use the ambient identity of the machine when running on Google Cloud. Provider credentials are envelope-encrypted at rest, like every other secret Yagra stores, and are never returned by the API after being set.

Requesting an explanation takes the Operator role (the same acknowledge-alerts permission as running an analysis). Generation is capped independently of the analysis runner:

• At most 2 generations run concurrently (YAGRA_RCA_MAX_CONCURRENT).
• At most 10 requests per minute (YAGRA_RCA_RATE_PER_MIN).
• Reports are cached for 15 minutes (YAGRA_RCA_CACHE_SECS), keyed on the incident’s evidence — asking twice about the same unchanged incident answers from cache instead of paying for a second inference. A regenerate option bypasses the cache when the situation has moved on, but never the rate limit.
• A single analysis is bounded by 6 tool-calling turns (YAGRA_RCA_MAX_TURNS), 240 seconds of wall clock (YAGRA_RCA_TASK_BUDGET_SECS), and a cap on total tool output. Hitting a bound returns the model’s last answer rather than failing the request. Setting YAGRA_RCA_MAX_TURNS=1 restores the pre-v0.1.23 single-shot behaviour exactly — no tools are offered, and the request sent to the provider is byte-identical to before.

The lookups run under the caller’s own visibility scope, so a group-scoped operator’s analysis cannot read a node they cannot see, and under a view-only allow-list: the write tools, run_analysis, run_rca, and the audit log are all out of reach, checked per folded branch rather than per tool. What it looked up is stored with the answer and replayed on both the WebUI and /mcp — for the same reason the evidence always was, that an explanation whose reader cannot check what it was based on is an assertion.

Settings ▸ AI also offers a connectivity test that sends a minimal request through the configured provider, so “is the key valid, can we reach the endpoint” is answered at configuration time rather than mid-incident.

A provider-side failure is reported as an upstream error (HTTP 502), distinct from a Yagra fault — so “the model refused” and “Yagra broke” are never confused in your monitoring of Yagra itself.

What leaves your boundary

Sending operational data to a model provider deserves an exact answer, so here it is.

Nothing is sent by default. With no provider configured, the feature is inert — no client exists and no request leaves the system. Data leaves only when an Admin has configured a provider and an Operator explicitly requests an explanation.

What the prompt contains, when one is sent:

• The root-cause node’s descriptive facts (name, kind, address, state).
• The alert being explained — what fired, when, on which metric or event.
• The node’s dependency context: up to 20 named dependents plus a count, and the chain of upstream ancestors.
• The cross-signal timeline around the incident — metric anomalies, passive events, and flow shifts.
• Up to 5 recent configuration changes from the audit log, since “what changed” is often the answer.
• Since v0.1.23, the results of any read-only tool the model chose to call — for example an interface’s traffic series, a node’s status, or matching syslog lines. This is the one part that is not fixed in advance: the model decides what to fetch, within the view-only allow-list, the caller’s visibility scope, and the turn/time/output bounds above. The transcript stored with the report is the exact record of what was retrieved, so what left is auditable after the fact rather than inferred. Set YAGRA_RCA_MAX_TURNS=1 to keep the prompt to the fixed evidence above.

What is never sent: monitoring credentials — the context is built by copying named fields, and a test pins that no credential-bearing structure can reach it; the tools available to the model are read-only and none of them return stored secrets. Device configuration bodies are excluded, and the fixed context still carries the shape of the incident rather than a raw export — but note that a tool call can now pull a specific metric series or event set on demand, which is the point of it. Device-originated text (like syslog messages) is fenced and framed as untrusted data, whether it arrives in the initial context or back from a tool, so a hostile log line is treated as evidence, not as instructions.

And it is observable. Every model call is counted in Yagra’s own Prometheus metrics — calls, tokens in and out, and errors, labeled by provider — so how often incident data leaves, and how much, is something you can graph and alert on, not something you have to take on faith.

If your boundary requirements are strict, the Vertex AI provider is the recommended shape: the same feature, with inference running inside your own cloud project and region.

Use it from an AI client

Everything on this page is also reachable over Yagra’s MCP tool surface: an AI assistant can run the same 15 analyses (run_analysis), poll a long run for its findings, and list recent jobs — under the same Operator permission and the same admission control as the WebUI. See MCP server.