High availability

A single core is a single point of failure: it schedules every check, evaluates every alert, and serves the API. High availability (HA) removes that by running more than one core against the same stores, with automatic leader election deciding which one is active. It is opt-in and off by default — a single-core deployment with HA disabled behaves exactly as before.

What HA covers

Yagra’s HA is an active/passive core pair (or more standbys) elected automatically:

• Cores — multiple yagra-core instances share the same PostgreSQL, NATS, VictoriaMetrics (and optional Redis and the rest). One is elected leader and does the work; the others stand by and take over within seconds if it fails. No double-polling, no duplicate notifications.
• Pollers — need nothing from this page. They are stateless and inherently redundant: run more than one poller per pool and a lost poller’s nodes fail over to the survivors automatically. See distributed pollers.
• The stores — are the platform’s job, not Yagra’s. PostgreSQL replication, VictoriaMetrics redundancy, and NATS clustering are configured with those products’ own tooling and are out of scope here; Yagra simply connects to whatever address you give it. Losing Redis is non-fatal regardless — it is a rebuildable mirror.

Put differently: HA makes the orchestrator survivable. The durability of the data underneath it is a property of how you deploy the backing services.

Enabling it

Set YAGRA_ENABLE_HA=true on every core instance, and point them all at the same stores. Leader election runs on a PostgreSQL advisory lock: the core holding the lock is the leader, and when it stops — crash, restart, upgrade — a standby acquires the lock and takes over within seconds. There is no extra election service to deploy; the database you already run is the arbiter.

The leader runs everything stateful-side: scheduling and poller assignment, passive-event ingest, and alert evaluation and notification. Standbys keep their API up and serve reads, while declining readiness (below) so traffic prefers the leader.

Two practical notes:

• Give each core a YAGRA_CORE_ID (e.g. core-a, core-b) so HA log lines and diagnostics say which instance they came from.
• The leader’s advisory-lock connection is held in addition to the normal pool (YAGRA_PG_MAX_CONNECTIONS, default 20 per core) — size PostgreSQL’s max_connections for all cores together, plus one.

The variables involved, all read per core:

Variable	Default	Role
YAGRA_ENABLE_HA	false	Opt into leader election; unset means no lock is even taken
YAGRA_CORE_ID	unset (generic label)	Names this instance in HA logs and diagnostics
YAGRA_SESSION_KEY_FILE	unset (per-core sessions)	Shared session signing key — see below

With YAGRA_ENABLE_HA unset, a single core behaves identically to a pre-HA deployment. Full variable details: the configuration reference.

Routing traffic

Two probe endpoints, with distinct meanings:

Endpoint	Answers	Meaning
/healthz	200 on every core, always	Process liveness — no auth, no store access. Use it for restart decisions.
/readyz	200 on the leader only; 503 on standbys	Readiness — “send traffic here”. With HA off, always 200.

Point your load balancer’s health check at /readyz and route API and WebUI traffic on it: exactly one core answers 200, so clients always land on the leader, and a failover redirects traffic automatically as the new leader starts answering 200. Keep /healthz for liveness probes — a standby is healthy but not ready, and restarting it for answering 503 on the wrong endpoint would defeat the purpose.

Each core also exposes a yagra_core_is_leader gauge on its Prometheus /metrics (1 on the leader, 0 on standbys) — worth wiring into your own monitoring so a failover is an event you see, not one you infer. See observability.

Sessions across cores

By default, login sessions are held per core process — which means a failover signs everyone out, and the new leader asks the whole team to log in again mid-incident.

The fix is a shared session-signing key. Mount the same 32-byte key into every core and point YAGRA_SESSION_KEY_FILE at it; session tokens are then stateless and signed, so a login is accepted by every core and survives a core restart or failover. Logging out, disabling a user, changing a role, or resetting a password still takes effect across all cores right away.

# generate once, then distribute the same file to every core (mounted read-only)
head -c 32 /dev/urandom > session.key

The key file must be exactly 32 raw bytes (or 64 hex characters). Handling is fail-closed: if YAGRA_SESSION_KEY_FILE is set but the file is missing, unreadable, or the wrong size, core refuses to start rather than silently minting sessions with a broken key. If HA is enabled without a session key, core starts and only warns — everything works, but failover logs users out.

Trying it locally

The repository ships a ready-made two-core overlay, docker-compose.ha.yml, which layers a second core (core-b, API on http://localhost:8081) onto the single-node stack and enables HA on both:

docker compose -f docker-compose.yml -f docker-compose.ha.yml up --build

Then watch the election from another shell:

# exactly one core is ready (200), the other is a standby (503):
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/readyz   # core-a
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8081/readyz   # core-b

# see who won the election ("acquired leadership" in the winner's log):
docker compose logs core core-b | grep -i leader

# stop whichever core holds leadership and watch the standby take over:
docker compose stop core
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8081/readyz   # 200 within seconds

The overlay is for evaluation. A production HA deployment is the same ingredients behind a real load balancer — for example a Kubernetes Deployment of cores with /readyz as the readiness probe, so only the leader is ever Ready. The deployment paths themselves are covered in the installation guide.

Limits

• Active/passive, not active/active. One leader does all scheduling, ingest, and alerting; standbys add availability, not throughput. Scale monitoring capacity horizontally with pollers, not with extra cores.
• Writes concentrate on the leader. Standbys serve reads and wait; routing on /readyz keeps clients pointed at the core that can do everything.
• Failover is seconds, not zero. In-flight requests during the takeover window can fail and should be retried; with the shared session key configured, no one has to log back in.