Observability

A monitoring system that cannot be monitored is a blind spot with a dashboard. Every Yagra binary — the core and every poller — therefore emits three signals out of the box: structured logs on stdout, a Prometheus /metrics endpoint, and (opt-in) OpenTelemetry traces. This page covers each signal, plus the built-in Settings ▸ System health page that turns them into a verdict for you.

Yagra monitors itself

The self-observability surface, at a glance:

Signal	Core	Poller	Default
Structured logs (stdout)	✔	✔	On (`info` level)
Prometheus `/metrics`	✔ — on the API port	✔ — on port `9100`	On
Health probes `/healthz` / `/readyz`	✔	—	On
OpenTelemetry traces	✔	✔	Off — opt-in via `YAGRA_OTEL_ENDPOINT`
Host resources (CPU/load/memory/disk)	✔	✔	On — charted under Settings ▸ System health

Nothing here requires extra containers: the single-node stack is fully observable with docker compose logs and a scrape of two HTTP endpoints. Tracing is the only part that needs a collector, and it is off until you point it at one.

Prometheus metrics

Core serves /metrics on the API port — 8080 in the bundled compose files — alongside the REST API and the health probes. No extra port to publish; anything that can reach the API can scrape the core.

Each poller serves its own /metrics on a fixed 0.0.0.0:9100. The default single-node compose does not publish this port on the host; it is reachable when the poller runs with host networking (the remote-site poller composition does) or natively. See Ports & firewall for the full port inventory.

A few representative series, to give a feel for what is exposed:

Metric	What it tells you
`yagra_core_is_leader`	`1` on the active core, `0` on a standby — the high-availability leader gauge, the same signal `/readyz` answers with
`yagra_mcp_tool_calls_total{tool,outcome}`	MCP tool invocations, per tool and outcome, when the MCP surface is enabled
`yagra_forward_dropped_total{reason}`	Forwarding records dropped, by reason (`queue_full`, `rate_limit`, `circuit_open`, …) — the first place to look when a relay destination falls behind
`yagra_llm_calls_total{provider,outcome}`	AI root-cause analysis calls per provider — billed external calls, worth alerting on
`yagra_core_backfill_results_total`	Poll results imported through the store-and-forward replay path after a network partition

This is a sample, not a catalog — the set grows with each feature. Scrape the endpoints to see everything the running version exposes.

Health probes

Two unauthenticated probes on the core’s API port separate “is the process alive” from “should traffic go here”:

GET /healthz — liveness. Always answers 200 ok on any running core, including a high-availability standby. It touches no backing store, so a database outage does not make your orchestrator restart a perfectly healthy process. Use it for container restart policies.
GET /readyz — leadership readiness. Answers 200 only on the core that currently holds leadership; a standby answers 503. Point a load balancer’s health check at it and traffic follows the leader through a failover — see High availability.

In a single-core deployment (HA off, the default) there is no standby to distinguish: /readyz is simply 200 whenever the core is up.

The core container image uses a real health check built on this, and the bundled compose files start the web container only after the core is serving — so a cold start never shows first-request errors.

The System health page

Settings ▸ System health is the WebUI’s answer to “is Yagra itself OK”, and it renders the server’s own verdict rather than re-deriving one in the browser:

Backing-service reachability — one row per store (PostgreSQL, Redis, VictoriaMetrics, and the optional VictoriaLogs and ClickHouse) plus the bus, with an aggregate “All reachable” / “Degraded” badge computed server-side. If a future dependency is forgotten by the page, it disagrees with the badge visibly instead of silently reading healthy.
Poll-loop health — scheduler and ingest counters, including how pools are being served (working sets versus the legacy per-job fallback, which turns amber when a pool has no live registered poller).
Host resources — CPU, load, memory, and disk charted over time for the core and every poller, so you can watch resource pressure build rather than only seeing a snapshot.

Which filesystems the disk charts cover is set per process with YAGRA_DISK_WATCH_PATHS — a comma-separated list of path or path=alias entries (default /=root), where the alias becomes the series label. Details in the configuration reference.

Logs

Both binaries write structured logs (the Rust tracing format) to stdout, so the container runtime owns retention and shipping:

docker compose logs -f core
docker compose logs -f poller

Verbosity is filtered with the standard RUST_LOG variable (default info), including per-component directives:

RUST_LOG=yagra-core=debug,yagra-poller=debug

Log lines never contain credentials — SNMP communities, SNMPv3 credentials, and API tokens are redacted from logs, API responses, and metric labels alike.

Logs on disk

Reading docker logs needs a shell on the host, which is exactly what a locked-down deployment does not grant — so a panic or an OOM would leave nothing retrievable. Core therefore also writes hourly JSON-lines files to YAGRA_LOG_DIR, in addition to stdout, keeping YAGRA_LOG_RETAIN_HOURS of them (default 48, pruned automatically). Put that directory on a named volume and the logs outlive the container, so a support bundle taken after a recovery still carries the run that died.

This is on by default in the shipped compose files; clear YAGRA_LOG_DIR in your .env to turn it off. Writes are non-blocking and drop rather than stall the poll loop, and an unwritable directory degrades to stdout-only with a warning instead of failing startup.

Pollers can opt in the same way, but no compose file mounts a directory for them: the support bundle carries core’s logs only. Poller heartbeat counters, poll-loop statistics, and host resources are in the bundle already.

Support bundle

When a deployment cannot be reached from outside, the hard part of diagnosing it is getting a complete picture out in one piece. Settings ▸ System health produces a downloadable support bundle — also available at GET /api/v1/system/support-bundle?since_hours=N — containing which binary is actually running (image source ref and build profile, not just the version), every system-health section, the allow-listed environment, applied migrations with their checksums, per-table sizes and connection counts, active alerts, the audit tail, core’s Prometheus scrape, and core’s own rotated log files.

It is built for a site where data does not leave casually, so the archive is designed to be reviewed before it is released: every entry is JSON or plain text, and MANIFEST.json lists what is carried and what is deliberately left out, with reasons.

Secrets are handled two ways. The environment is carried by an allow-list, because a deny-list of password-shaped names would miss the credential inside YAGRA_DATABASE_URL’s userinfo — the one that actually ships. Then every assembled byte is scanned, and a match aborts the export rather than redacting it, so a credential arriving through an unanticipated path is caught too. A refusal answers 500 support_bundle_redaction_failed naming the file and the rule, never the value.

Generating one requires ManageConfig + ManageCredentials + ViewAudit — all three, so it cannot become a way to read the audit log or the credential report through an endpoint whose name mentions neither. In practice that means an administrator.

Distributed tracing

Tracing is opt-in: set YAGRA_OTEL_ENDPOINT (or the standard OTEL_EXPORTER_OTLP_ENDPOINT) to an OTLP/HTTP collector and both core and poller export spans. Left unset, there is zero tracing overhead — logs only — so the single-node stack needs no collector.

What you get is a single trace per poll that crosses process boundaries: core’s dispatch span → the poller’s probe span → core’s result-ingest span, plus a span per northbound API request. The trace context rides the bus with jobs and results, is omitted from the wire when tracing is off, and is ignored by an older poller — so it stays safe during a rolling upgrade.

Try it locally. The bundled compose file ships a Jaeger profile:

docker compose --profile tracing up

Then uncomment YAGRA_OTEL_ENDPOINT: http://jaeger:4318 on both the core and poller services in docker-compose.yml, and open the Jaeger UI at http://localhost:16686.

At scale, sample. Tens of thousands of nodes polling on an interval would otherwise emit a trace per poll. Set:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.01

The parentbased_* samplers keep a whole trace’s sampling decision consistent across the core⇄poller hop, so you get 1% of complete traces rather than fragments.

In production, point the endpoint at an OpenTelemetry Collector that forwards to your tracing backend (Tempo, Jaeger, Honeycomb, …). A remote-site poller needs its own reachable collector endpoint — trace export does not travel over the NATS bus.