Monitoring and alerting

Most validator outages are found too late: the process is running, its health check says ok, and the node isn’t doing its job. This page lists the signals that catch real problems early, and the false alarms that train people to ignore pages.

Where the specifics come from

Log lines and metric names below were checked against Agave v4.4.0 source, and they apply to Jito-Solana as well. The host-level and Alpenglow-key sections apply to every client. For Firedancer metrics, start from its monitoring guide: a Prometheus endpoint, a monitor terminal UI and an optional browser GUI. Contributions that map these alerts onto Firedancer metrics are very welcome.

Principles

1. Alert on causes, not on the node’s opinion of itself. getHealth and similar checks are computed from data the node may not be receiving. A node that has lost all inbound traffic can look “healthy” for a long time, because nothing new tells it otherwise. “Zero inbound packets while the process is up” is the signal that catches it.
2. Separate “the process is up” from “the process is healthy”. Whether the log is still being written tells a dead or hung process apart from a live one that is isolated or stuck. Most stall alerts should only fire while the log is still active, or they fire on every shutdown.
3. Treat a restart as an episode, not as falling behind. A freshly started node is behind by definition. Page if it hasn’t caught up within N minutes of starting, not on “behind” during that window. Then send one “back in sync after X minutes” message.
4. Edge-trigger and debounce. Send one message when a condition starts, one when it clears, and at most a periodic reminder in between. Persist that state, so restarting the monitor doesn’t re-send every alert.
5. Watch the watcher. Have the monitor announce itself when it starts, which proves the alert path works. Add a second, independent check from outside the host, so the monitor dying, or the whole host dying, still pages someone.

Alert conditions

Severity: page means wake someone up, warn means look today, and info means dashboard only.

Opinion

The thresholds are starting points that have held up in practice, not constants. Tune them to your stake (how often you lead) and your tolerance for noise.

Liveness

Condition	Signal	Starting threshold	Severity
Validator unreachable	getSlot against the node’s RPC fails, checked from another host	2 min	page
Not confirming	The node’s confirmed slot isn’t advancing	2 min	page
Not rooting	The node’s finalized/root slot isn’t advancing. Agave logs <identity>: new root <slot> from votor.	5 min	page
Network isolated	Inbound turbine and gossip traffic both zero while the log is still active. In Agave, the shred_sigverify and gossip_receiver datapoints.	30 s	page
Restart loop	Repeated process starts (Agave logs listen_for_signals() has started once per start)	≥ 3 in 15 min	page
Restart not catching up	Time since start without catching up	~20 min	warn
Behind the cluster	The node’s latest replayed slot versus the cluster tip as seen from peers, not /health	sustained gap of more than a few slots	warn

• Not rooting: a stuck root also stops ledger cleanup, so the disk will eventually fill. It can happen while replay looks fine.
• Network isolated: require both turbine and gossip to be silent, because one path dropping out alone is noise. Leave repair traffic out: a trickle continues even when the node is isolated.
• Restart loop: usual causes are a bad flag, a full disk, or a slot that crashes replay.
• Restart not catching up: include the gap, the replay rate and an ETA in the message.

Consensus and block production

Condition	Signal	Starting threshold	Severity
Voter not participating	The consensus_pool_service datapoint’s certificates_skipped_unstaked counter is non-zero, or certificates_sent isn’t advancing	any, on the node that should be voting	page
BLS key mismatch	Agave: No authorized voter keypair matches rank-map BLS key for vote account …. Firedancer: … unable to vote.	any	page
Vote account underfunded	Vote-account balance against the admission minimum (rent-exempt plus the per-epoch VAT, see Key setup)	warn well ahead of the next epoch boundary	warn → page
Leader slots skipped	Your leader slots that end without a confirmed block	0 produced in your last N rotations, bounded in time	page
Dead slots	Agave replay-stage-mark_dead_slot datapoint	several in ~10 min, excluding expected reasons	warn
Bank-hash divergence	Your frozen hash for a slot differs from the cluster’s confirmed hash	any	warn; page if the root stays stuck behind it
Vote egress saturated	Agave error alpenglow transport egress channel full; dropping votes/certs!	any sustained rate	warn
Identity balance	Identity account balance, if your block-revenue collector is the identity	below your sweep threshold	info

• Voter not participating: the node skips sending certificates when its identity isn’t in the staked set. That’s expected on a hot spare, an RPC node or a new validator. On your active voter it means you’re caught up but not taking part.
• BLS key mismatch: the node runs but casts no votes. See Key setup.
• Leader slots skipped: count in rotations rather than slots. A low-stake validator may lead only a few times per epoch, so “last 8 slots” can span days.
• Dead slots: a single dead slot is background noise, and some reasons are expected. A leader abandoning its own block shows up as BlockAborted. Count only unexpected reasons.
• Bank-hash divergence: this usually heals itself, because the node dumps and repairs the block. Rooting past the diverged slot is the definitive sign it has healed.

Tower-era checks don’t carry over

Under Alpenglow, votes are off-chain BLS messages: there is no on-chain “last vote”, no vote credits and no vote transaction fees. Any check built on a vote account’s last vote, tower lockouts, vote credits or identity vote-fee balance needs replacing. The participation, BLS key and admission signals above take their place.

Host-level concerns

Memory

• Alert on MemAvailable, never MemFree. On a validator almost all free RAM is page cache, so MemFree sits near zero on a healthy machine.
• Use two triggers, each sustained for about a minute:
  • a level trigger, for example below 15% warn and below 5% page;
  • a trend trigger: fit the last ~5 minutes and warn when exhaustion is projected within 30 minutes, page within 15.
• On a large-RAM machine the trend trigger buys the most time, because the level threshold is often crossed only minutes before the out-of-memory kill.
• Only evaluate the trend below roughly 40% available. Otherwise memory settling onto its normal plateau after a restart extrapolates to zero, and pages with half the RAM free.
• Put the rate in the message (“falling 3 GB/min, empty in ~29 min”). A level alone isn’t something anyone can act on.

Don’t run heavy scans against your validator’s own RPC

Long full-accounts scans (getProgramAccounts without a narrow index, large token-account scans) against a voting validator’s RPC can hold up accounts cleanup and flushing, sometimes hours later. The write cache then grows until the host runs out of memory. Run scans against a separate RPC node or a provider, and keep the validator’s RPC private.

Disk

• Watch capacity on every volume: ledger, accounts, snapshots, logs, and the monitor’s own data. For example, warn at 85% and page at 95%.
• In summaries, rank “disk full” above “stalled” or “behind”: a full disk is usually the cause of a stall, not a symptom of one.
• Common causes of a filling disk:
  • a root that isn’t advancing, since ledger cleanup only purges below the root;
  • snapshots piling up;
  • logs without rotation;
  • monitoring data kept with no retention limit.
• Graph I/O rates, but alert on capacity.

Network

• Graph receive and transmit traffic and drops per interface. When you sum interfaces, don’t double-count bond or bridge member interfaces, or tunnels whose traffic also appears on the physical NIC.
• Alert on inbound silence (see “Network isolated” above). NIC drops belong on a graph, not a pager.

Clock

• Keep NTP or chrony synchronized (validator-ansible: time sync), and alert when it’s unsynchronized or the offset goes beyond ~100 ms.
• If you measure delivery latency per leader from local data, compare each leader against the window’s median. Your own clock offset then cancels out.

CPU and limits

• Watch for frequency throttling (CPU governor not set to performance, or thermal limits) and for saturated replay or scheduler threads. These are graphs for investigating, not pages.
• Check the file-descriptor limit, vm.max_map_count and the UDP buffer sysctls once at startup. Agave logs warnings for several of these.

Host tuning itself (governor, sysctls, limits) is covered by the validator-ansible template.

False alarms to design out

Each of these has paged real operators.

1. /health says ok right after a restart while far behind. Agave compares against the newest confirmed slot stored in its own blockstore, which survives the restart, so the reference point is where the cluster was before shutdown. Don’t page on, or announce, /health changes while the node is starting up.
2. “Behind by 0” right after a start. If “caught up” requires the gap to stay low for a while, the node reads “not caught up” at gap 0 during that time. Page “behind” only once the gap itself exceeds the threshold.
3. A balance that reads 0 now and then. getBalance with default commitment answers from the finalized bank. On a node that’s catching up, that bank can be old enough to return 0. Check context.slot in the response, discard readings from a bank far behind the tip, and debounce.
4. Dead slots that are normal protocol behavior. Leaders abandoning their own blocks produce dead slots with expected reasons. Exclude those.
5. “Isolated” during startup. “Last packet seen” timestamps carried over from before a restart make a node loading its snapshot look isolated. Reset them on every start, and stay quiet until the first packet arrives.
6. The memory trend extrapolating a plateau. See the 40% gate above.
7. Fake divergences from half-written log lines. A log tailer that treats an unterminated last line as complete parses truncated hashes. Hold partial lines until their newline arrives.
8. A healed divergence escalated again later. Only escalate a divergence while the root hasn’t passed it.
9. Planned restarts paging the external check. Either accept the page, or grade hot spares and secondary nodes as warn: a spare being down reduces redundancy, but it isn’t an outage. Silencing windows are easy to forget to close.
10. “Not receiving” on a feed the host never subscribed to. For opt-in feeds (for example multicast shred feeds), alert on no traffic only if the host is configured to receive.