The short version. On 8 August an agave validator went into a crash loop after a routine rebuild. The cause was real and the fix took twenty minutes. Proving the fix had worked took most of the day, because the three tools I reached for to verify AF_XDP all report off on a node where it is working correctly. Ten days later I made the same class of mistake again, in an entirely different way, and briefly concluded that a log line had been removed from the client. It had not. It differs from the string in my own runbook by one capital letter, and every health check I had written against it had been silently doing nothing for weeks.
Both times the system was fine. Both times the instrument was wrong. That pattern is worth more than either individual gotcha, so here is all of it — with the wrong outputs, which are the valuable part.
Scope. This is the Alpenglow community test cluster, running Ashwin Sekar’s agave 4.3.0 fork, with XDP on the transmit path via --xdp-interface / --xdp-cpu-cores and zero-copy off. Mainnet agave uses different flags and a different code path, and the required capability set changes with zero-copy (§3). Read this as one operator’s incident, not as a drop-in guide for a mainnet node.
1. The crash loop
The node runs agave with AF_XDP enabled on the transmit path:
--xdp-interface <IFACE> --xdp-cpu-cores <N-M>The rebuild used the standard install path for this fork:
CI_COMMIT=$(git rev-parse HEAD) scripts/cargo-install-all.sh \
--validator-only ~/.local/share/solana/install/active_releaseRestart, and:
ERROR ... the current configuration requires the following capabilities,
which have not been permitted to the current process:
[CAP_NET_ADMIN, CAP_NET_RAW]Fatal in about a second, then activating (auto-restart) forever.
This part of the system is honest and loud. If the XDP flags are present and the capabilities are not, the process dies immediately and says exactly why. No ambiguity, no silent degradation. It is the only part of the day that behaved well.
2. The fix, and the wrong instinct
The obvious move is setcap cap_net_admin,cap_net_raw+ep on the binary. Don’t.
cargo-install-all.sh replaces the binary file on every build. File capabilities live on the inode, so every rebuild silently wipes them. That was the actual root cause of the crash loop: a grant made at some earlier point had been erased by the build I had just run, and I had no memory of ever setting it.
The mechanism that survives rebuilds is a systemd drop-in granting ambient capabilities to the service:
# /etc/systemd/system/validator.service.d/override.conf
[Service]
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAWsystemd grants these at exec time, so they do not care what happened to the file on disk.
There is a gotcha inside the gotcha. systemctl edit opens a scratch buffer containing the line ### Edits below this comment will be discarded. Content placed below that marker is thrown away — but the drop-in file is still created, so systemctl status cheerfully lists it under Drop-In:. It looks configured. It does nothing. That cost a diagnostic cycle on its own.
Confirm systemd actually holds the setting before restarting:
systemctl show validator -p AmbientCapabilities
# must echo: AmbientCapabilities=cap_net_admin cap_net_raw3. Three instruments, all wrong
With the drop-in in place the node came up clean. Now I wanted to confirm XDP was actually working. This is where the day went.
Four of the five capability fields read zero. The standard check:
grep Cap /proc/$(systemctl show validator -p MainPID --value)/status
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000Every field that describes a capability the process actually holds — inheritable, permitted, effective, ambient — reads zero. I took that as proof the grant had failed and went back to fighting systemd.
It was not. agave drops its capabilities after initializing XDP. Once the AF_XDP sockets exist the process no longer needs CAP_NET_ADMIN, so it gives them up. Good security hygiene, and completely invisible to anyone reading /proc afterward. Zeros are the correct reading on a healthy node.
The one non-zero field is the trap in the other direction. CapBnd is the bounding set — a ceiling on what the process could ever acquire, not a record of what it has. It stays full because nothing dropped it, and it would look identical on a node that never had the capabilities at all. Read in isolation it says “capabilities are fine here,” which is just as uninformative as the four zeros above it, and wrong in the more dangerous direction.
The NIC shows no XDP program. So I went a layer down:
ip link show <IFACE> # no xdp/prog stanza
sudo bpftool net show # nothing XDP-relatedEmpty. More proof of failure, surely.
No — but the reason is not the one I assumed at the time, and the difference matters. I concluded that agave’s transmit path uses AF_XDP sockets without attaching any eBPF program, so there was simply nothing for these commands to find. The source says otherwise. xdp/src/transmitter.rs does load and attach a program, in driver mode, and deliberately keeps it attached for the life of the process. But it does so only when zero-copy is enabled:
let maybe_ebpf_result = if zero_copy {
let _ebpf_caps = CapGuard::raise([CAP_BPF, CAP_PERFMON])
.expect("raise ebpf capabilities");
Some(load_xdp_program(&dev)
.map_err(|e| format!("failed to attach xdp program: {e}")))
} else {
None
};This node runs without zero-copy, so no program is loaded and bpftool net show reports an empty xdp: section — confirmed again while fact-checking this piece. The same conditional explains something else that had puzzled me: the crash-loop error named two capabilities, while agave’s own CHANGELOG lists four for XDP. validator/src/commands/run/execute.rs requires CAP_NET_ADMIN and CAP_NET_RAW whenever XDP is configured, and adds CAP_BPF and CAP_PERFMON only if xdp_config.zero_copy — because that is the branch that loads the program. One conditional, two puzzles.
Be precise about what this means for the check. ip link and bpftool are uninformative in this configuration, not universally. On a zero-copy node they would show the attached program and would be a genuine signal. Which is exactly the mistake this article is about: I generalised from one node’s behaviour to a property of the implementation, and I was wrong. I found that out by reading the source, after publishing the first version of this paragraph.
The logs “confirmed” it was running. Meanwhile I grepped the log for XDP and found matches, which briefly confused things in the opposite direction. They were block IDs and hashes — ...86XDPDoP5r, ...fuXDP4nVNP3.... Base58’s alphabet makes any three-character token a coin flip in a validator log. So one instrument said definitely off, another said apparently on, and both were noise.
For completeness: I also chased a nosuid mount hypothesis to explain the file-capability failure. Also wrong — the binary sits on an ordinary rw,relatime ext4 mount. Three hypotheses, three dead ends, every one of them generated by trusting a reading.
4. What actually works
After all that, here is the whole of what means something.
Positive ground truth — AF_XDP sockets bound to the NIC queues:
sudo ss -f xdp
# healthy: one row per configured queue, e.g. <IFACE>:q0 and <IFACE>:q1This is the check. Everything else is supporting evidence.
Supporting — flags present in the live process, and the process is alive:
tr '\0' '\n' < /proc/$(systemctl show validator -p MainPID --value)/cmdline \
| grep xdpThe reasoning is what makes this sufficient, and it is worth stating explicitly: the capability check is fatal. A process running with XDP flags in its command line necessarily had the capabilities when it started. Survival is the proof.
Failure signal — the capability error at a fresh timestamp:
grep "requires the following capabilities" ~/solana-validator.log | tail -2
systemctl status validator --no-pager | head -8 # activating (auto-restart)?Note fresh. An old error sitting in a rotated log will happily convince you of a problem you fixed hours ago.
Uninformative here: the capability fields in /proc/<pid>/status — on any agave node running XDP, because the process drops what it no longer needs — and ip link / bpftool net show on a node without zero-copy, because nothing is attached to find. Enable zero-copy and the latter two become a real check. The distinction is the point: an instrument is not uninformative in general, it is uninformative against a particular configuration, and you have to know which one you are pointing it at.
5. Ten days later, the same mistake
On 18 August I was revising the runbook and ran a check it described as ground truth for VAT status:
grep "VAT Health check" ~/solana-validator.logNothing. I checked six days of rotated logs. Nothing there either — while a case-insensitive search for VAT returned nearly six thousand hits in a single day’s log. Rotation was ruled out. The obvious conclusion was that agave 4.3.0 had removed or renamed the line, and I wrote that down as a finding.
It was wrong. One command settled it:
grep -io "vat[a-z ]*" ~/solana-validator.log | sort | uniq -c | sort -rn | head
1754 VAT checks
1754 VAT Health CheckVAT Health Check. Capital C. My runbook had documented it with a lowercase c. Every case-sensitive grep against that string had silently returned nothing, for however many weeks the typo had been sitting there, while the validator emitted the line every thirty seconds without interruption:
[2026-08-18T14:38:52Z INFO solana_core::replay_stage] VAT Health Check: Passing local
VAT checks.
Assuming you are in the top 2000 of stake, you will be included in voting and
block building in epoch 97A health check that had been quietly doing nothing, and an operator reading its silence as good news. Same failure as the capability zeros, in a different costume.
Two incidental findings from that line, neither of them the point. The “top 2000 of stake” clause does not bind on a cluster running ~114 validators — it is inherited mainnet-oriented boilerplate. And the named epoch runs two ahead of current, against a one-epoch VAT lookahead, which suggests the message is forward-looking by a further epoch. Neither is a problem here; both are the kind of thing you only notice once you can actually read the line.
6. The pattern
Both incidents share a shape: an instrument reported nothing, and I read nothing as evidence of absence.
The capability fields report zero on a healthy node because the process dropped them on purpose. ip link reports empty on a healthy node because this implementation attaches no program. The grep returns empty on a healthy node because the string has a capital letter in it. In all three cases the reading on a working system is identical to the reading on a broken one — which means the instrument carries no information, and I was building conclusions out of noise.
The discipline that would have saved the day is one question, asked before trusting any check:
What would this command output if the system were healthy?
If the answer is “the same thing it is outputting now,” stop. You do not have a diagnostic. You have a coin flip.
The corollary is the more useful half, because it applies to systems that are not currently on fire: a monitoring check that has never fired is not evidence of health. It is equally consistent with a check that cannot fire. Every grep-based alert in a fleet is one upstream string change away from becoming a decorative no-op, and nothing in the system will tell you when that happens. It is worth auditing them on that basis rather than waiting for the incident that reveals it.
7. Practical takeaways
- Grant XDP capabilities via a systemd
AmbientCapabilitiesdrop-in, neversetcap— file capabilities do not survivecargo-install-all.sh. - Verify with
sudo ss -f xdp, which works in either mode./proccapability fields are uninformative because agave drops caps after setup;ip linkandbpftoolare uninformative only without zero-copy, since the eBPF attach sits behind that flag. - Confirm drop-ins with
systemctl show <unit> -p AmbientCapabilities, becausesystemctl editwill silently discard misplaced content while still appearing configured. - Never grep validator logs for short bare tokens — base58 will match them by coincidence.
- Before concluding a log line is gone, check case-insensitively:
grep -io "<token>[a-z ]*" <log> | sort | uniq -c | sort -rn. - Anchor log checks on module paths such as
solana_core::replay_stagerather than message text, which changes more often. - Audit any grep-based health check that has never fired. It may not be healthy. It may be looking for a string that does not exist.
The full operational runbook for this node — upgrade drill, regenesis procedure, complete incident history — is published as a dated snapshot here. If any of this behaves differently on your hardware or your agave revision, I would like to know.