Deploying rustbgpd
Install, operate, and upgrade rustbgpd.
Install, operate, and upgrade rustbgpd.
This is the end-to-end operator runbook: install, run, validate,
reload, observe, and upgrade. It's intentionally smaller than the M-
series interop matrix in INTEROP.md — that document
proves rustbgpd interops cleanly with FRR / BIRD / GoBGP, this one gets
your daemon onto your box.
For deeper references:
CONFIGURATION.md— every config field, with examples.reload-matrix.md— which fields hot-apply vs. need a restart.OPERATIONS.md— debugging, log filtering, failure modes.SECURITY.md— firewalling, gRPC mTLS, authorization tiers.
Status & expectations
rustbgpd is public alpha overall. The exception is the narrow,
machine-pinned v1 route-server / route-reflector / controller contract,
which covers only its listed config fields, native gRPC signatures, CLI/JSON
contracts, and control-plane roles. Unlisted TOML and API surfaces can still
change between minor versions. See CHANGELOG.md for
migration notes and run rustbgpd --check <new config> against the new binary
before swapping it in. FlowSpec controllers should also follow the
controller upgrade and rollback procedure
to verify server outcomes and view acknowledgements, then reconcile desired
rules after restart.
The supported daemon targets are Linux x86_64 and aarch64. Published binaries use a glibc 2.31 baseline. See the canonical platform support contract for CI and non-Linux boundaries.
Suitable for: lab pilots, data-center fabric pilots, IX route-server pilots, automation-heavy control-plane work where the API surface evolution is acceptable. Not yet suitable for fully unattended production deployments where the operator can't react to a CHANGELOG note.
Install
Verified installer
The installer ships as the install.sh release asset from v0.70.0 onward
(earlier releases do not include it) and is also available in a source
checkout as packaging/install.sh. It resolves
latest once, then downloads
the selected artifact and its matching per-architecture checksum manifest from
that same tag. It refuses unknown architectures, non-glibc hosts, glibc below
2.31, missing checksum rows, duplicate checksum rows, and checksum mismatches
before installing.
On Debian/Ubuntu it installs the verified native .deb; on RHEL, Rocky, and
AlmaLinux 9+ it installs the verified native .rpm. The package keeps the
existing non-replacing configuration behavior. The installer never enables or
starts a service; edit the configuration and choose that step yourself.
For a reproducible install from this checkout, inspect and run the installer
with an explicit stable tag (vMAJOR.MINOR.PATCH):
less packaging/install.sh
sh packaging/install.sh --tag v0.83.0Without a source checkout, fetch the published asset:
curl -fsSL https://github.com/lance0/rustbgpd/releases/latest/download/install.sh | shOther GNU/Linux distributions must use an explicit prefix or download-only
mode. --prefix extracts the verified tarball's existing layout directly into
an empty directory; it does not create users, write configuration, or install
systemd units. --download-only writes the verified selected artifact and its
manifest without installing either.
sh packaging/install.sh --tag v0.83.0 --prefix /opt/rustbgpd-0.83.0
/opt/rustbgpd-0.83.0/rbgp doctor
sh packaging/install.sh --tag v0.83.0 --download-only ./rustbgpd-v0.83.0Pre-built binary tarball
Tagged releases publish rustbgpd-linux-amd64.tar.gz and
rustbgpd-linux-arm64.tar.gz under
GitHub Releases. Each
ships rustbgpd (the daemon), rbgp (the CLI),
rs-config-render (the route-server config renderer), and
birdwatcher-adapter (the Alice-LG shim), plus man pages
and shell completions under share/:
rustbgpd, rbgp, rs-config-render binaries
birdwatcher-adapter Alice-LG birdwatcher shim
LICENSE-MIT, LICENSE-APACHE, LICENSES.md licenses
rustbgpd.schema.json config JSON Schema
share/man/man1/rbgp.1 CLI man page
share/man/man8/rustbgpd.8 daemon man page
share/completions/rbgp.{bash,zsh,fish} shell completions
share/systemd/rustbgpd.service hardened systemd unit
share/systemd/rustbgpd@.service opt-in per-handle systemd unit
share/systemd/rustbgpd-dataplane.conf opt-in CAP_NET_ADMIN drop-in
share/monitoring/rustbgpd-overview.json Grafana overview dashboard
share/monitoring/rustbgpd-evpn.json Alpha EVPN Grafana dashboard
share/monitoring/rustbgpd-alerts.yml Prometheus alert rules
share/monitoring/rustbgpd-alerts_test.yml promtool rule testsThe binaries are built against glibc 2.31; any distro at or above it works (Debian 11+, Ubuntu 22.04+, RHEL/Rocky/Alma 9+).
The filename is the same on every release; releases/latest/download/
always resolves to the current tag, so this snippet never needs a
version bump.
# Pick the right arch.
SUFFIX=linux-amd64 # or linux-arm64
TARBALL=rustbgpd-${SUFFIX}.tar.gz
curl -fL -o "$TARBALL" \
"https://github.com/lance0/rustbgpd/releases/latest/download/${TARBALL}"
curl -fLO \
"https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$TARBALL" '$2 == file || $2 == "./" file { print }' \
"checksums-${SUFFIX}.txt" | sha256sum -c -
tar -xzf "$TARBALL"
sudo install -m 0755 rustbgpd rbgp rs-config-render birdwatcher-adapter /usr/local/bin/
sudo install -m 0644 share/man/man1/rbgp.1 /usr/local/share/man/man1/
sudo install -m 0644 share/man/man8/rustbgpd.8 /usr/local/share/man/man8/
sudo install -m 0644 share/completions/rbgp.bash \
/usr/share/bash-completion/completions/rbgp
sudo useradd --system --home-dir /var/lib/rustbgpd \
--shell /usr/sbin/nologin rustbgpd 2>/dev/null || true
sudo install -m 0644 share/systemd/rustbgpd.service \
share/systemd/rustbgpd@.service /etc/systemd/system/
sudo install -d -m 0755 /etc/rustbgpd
rustbgpd --init-config edge | \
sudo tee /etc/rustbgpd/config.toml >/dev/null
sudo chmod 0640 /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml
# Kernel-dataplane hosts only; control-plane-only hosts omit this drop-in.
sudo install -d /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 share/systemd/rustbgpd-dataplane.conf \
/etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.confThe man pages and completions are also generated on demand by the
binaries themselves (rbgp man, rustbgpd --man,
rbgp completions bash|zsh|fish), so an installed binary can always
regenerate them. Bash and Zsh complete flags at every command level. Fish
completes flags only for top-level commands and their direct subcommands;
for deeper commands such as rbgp policy chain set-import, it completes the
subcommand name but not its flags. All three shells offer --json-lines,
--pager, and --json-version only on command paths where those flags apply.
To pin to a specific tag for reproducibility, swap latest for the
version, e.g. releases/download/v0.83.0/${TARBALL}. SHA-256
checksums are published alongside each tarball as
checksums-${SUFFIX}.txt.
Verify:
rustbgpd --version
rbgp --version
rs-config-render --versionrs-config-render matters only if you run an IXP route server — it turns
arouteserver template-context output into rustbgpd configuration
(tools/rs-config-render/). It is
in the archive either way, so install it now rather than discovering it is
missing from a cron refresh later.
For an IXP Manager candidate, run the per-handle
rustbgpd@.service instance (its ExecStart reads
/var/lib/rustbgpd/<handle>/activation/current/config.toml), then let the
renderer publish and settle an immutable generation under that handle:
sudo -u rustbgpd /usr/local/bin/rs-config-render activate \
--router-handle rs1-ipv4 \
--candidate /var/lib/rustbgpd/ixp-manager/candidate \
--runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
--state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
--host-state-dir /var/lib/rustbgpd/ixp-manager-host \
--check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
--rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
--activation-command /usr/bin/sudo \
--activation-arg=-n --activation-arg /usr/bin/systemctl \
--activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4Render and activate as the rustbgpd service identity; it must own the private
candidate, the per-handle runtime and activation directories, and the shared
host-state directory. The runtime directory basename must equal the handle, the
activation directory must be exactly <runtime>/activation, and --rbgp-addr
must be that runtime's grpc.sock. Authorize that account in sudoers for only
the literal /usr/bin/systemctl reload-or-restart rustbgpd@rs1-ipv4. Pre-create
the absolute runtime, activation, and host-state directories as non-symlink
mode-0700 directories owned by rustbgpd; do not let another process publish or
reload during the call. Add --initial only when
both no current generation and no reachable daemon exist. Normalized comparison
TOML is limited to 4,194,299 bytes (4 MiB minus five encoded-request bytes).
The activation executable must be synchronous. Exit 7 means the candidate was
not applied: the executable could not start, or the daemon rejected its reload
without runtime effect. The prior link is then restored without another
activation and the prior runtime is verified unchanged. The helper accepts a
rejection only when rbgp metrics, read before the executable and again after
the prior link is restored, shows the same daemon process recorded exactly one
SIGHUP outcome, rejected_no_effect, and no runtime-config settlement is in
progress. Otherwise, once the executable starts, a nonzero exit, timeout, or
unsettled result leaves current on the candidate and returns exit 5; a
rejection that cannot be re-proven after the prior link is restored also returns
exit 5, with current on the previous generation. Leave
retained state untouched and inspect the current private receipt if present;
recovery or receipt durability is unproven, and the receipt may be absent or
stale when its final write or directory sync failed. rs-config-render status
reads that state without changing it and the rs-config-render recover verbs
(keep-current, rollback, release-lock, clear; dry run unless --apply)
resolve it; the operator procedure for exit 5 is the
activation manual-recovery runbook.
To drive the same path directly from IXP Manager v7.4, create a separate
mode-0700 candidate directory for the run and store the API key in an absolute
mode-0600 regular file owned by rustbgpd:
sudo -u rustbgpd /usr/local/bin/rs-config-render ixp-manager-lifecycle run \
--ixp-origin https://ixp.example.net --router-handle rs1-ipv4 \
--api-key-file /var/lib/rustbgpd/ixp-manager/api-key \
--candidate-dir /var/lib/rustbgpd/ixp-manager/candidate-1 \
--runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
--state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
--host-state-dir /var/lib/rustbgpd/ixp-manager-host \
--max-prefix-restart-seconds 300 \
--check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
--rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
--activation-command /usr/bin/sudo \
--activation-arg=-n --activation-arg /usr/bin/systemctl \
--activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4The helper uses the exact v7.4 lock, Foil configuration, updated, and release
endpoints. HTTPS platform roots are required; redirects, proxies, URL
credentials, and path-prefixed origins are disabled. Requests send the API key
only in the credential header. A synced private journal precedes each remote
effect. Exit 6 means one callback is pending and may be retried with
ixp-manager-lifecycle resume; exit 5 means acquisition or activation remains
uncertain and no callback is attempted automatically — see the
activation manual-recovery runbook.
Delivery is at-least-once.
Bound retained activation generations
Every distinct activation leaves an immutable generation under
<state-dir>/generations/; activation and lifecycle commands never remove
one. Run rs-config-render prune --keep N separately. It reports a dry-run
plan unless --apply is present.
| Retention reason | Protected generation |
|---|---|
current | The current symlink target |
receipt | The last activation receipt's candidate |
predecessor | The last activation receipt's previous generation |
journal | A generation whose render receipt is named by a pending lifecycle journal |
keep-N | The N most recently published generations; --keep 0 keeps only referenced generations |
Pruning refuses with exit 2 while a host fence or competing host command exists, or when retained receipt or journal state cannot be read safely. The host lock remains held through removal, so recovery state cannot appear between planning and deletion. The tool README's pruning section is authoritative for the complete command, stable output, and cron-safe pattern.
rs-config-render exit codes
Every rs-config-render subcommand shares one exit-code table, mirrored from
the tool README:
| Exit | Meaning |
|---|---|
| 0 | success — rendered, candidate validated, activated or no-op, or updated delivered |
| 1 | invalid input — the context or IXP Manager document is unreadable, unparseable, or carries an unknown field, or a site-local file is unreadable (also the generic failure when the final stdout line cannot be written) |
| 2 | refused — an unsupported knob, an invalid option combination, an unmet precondition (including an unavailable strict checker), no upstream lock acquired, or a definite pre-activation refusal released; nothing is published or activated and no generation, receipt, or journal is left behind (the lifecycle folds a strict-check rejection into this code and leaves that candidate, receipt-less, in its candidate directory for inspection) |
| 3 | aborted — a generated set is empty or under the plausibility floor (arouteserver mode) |
| 4 | shape drift — the context's top-level structure drifted from the pinned fingerprint; pass --allow-shape-drift to proceed (arouteserver mode) |
| 5 | manual recovery — a human is needed: the activation effect is uncertain (current stays on the candidate, or on the previous generation when a no-effect rejection could not be re-proven after restoring it) or a recover --apply step did not complete (current is wherever that step left it — on the rollback target after a rollback that did not settle); retained state and any upstream lock are kept and no callback is issued; inspect with status before acting |
| 6 | callback pending — one durable updated or release callback is undelivered; run ixp-manager-lifecycle resume |
| 7 | rolled back — the candidate was not applied: the activation command never started, or the daemon rejected its reload without runtime effect; the prior generation is restored and proven and the lock is released; retrying is safe |
| 8 | output unusable — the candidate directory is not an absent or empty private directory (IXP Manager mode), could not be created or written (arouteserver mode), or a prune --apply removal failed |
| 9 | strict check failed — rustbgpd --check --strict ran and rejected the rendered IXP Manager candidate (the only path to this code); its files stay in the candidate directory without a receipt |
Debian / RPM packages
Tagged releases also publish native packages built with
nfpm from the same binaries as the
tarballs (packaging/nfpm.yaml):
rustbgpd_X.Y.Z_{amd64,arm64}.deb— Debian 11+, Ubuntu 22.04+rustbgpd-X.Y.Z-1.{x86_64,aarch64}.rpm— RHEL/Rocky/Alma 9+
# Debian / Ubuntu
curl -fLO https://github.com/lance0/rustbgpd/releases/latest/download/rustbgpd_X.Y.Z_amd64.deb
sudo apt-get install -y ./rustbgpd_X.Y.Z_amd64.deb
# RHEL / Rocky / Alma
sudo dnf install -y \
https://github.com/lance0/rustbgpd/releases/download/vX.Y.Z/rustbgpd-X.Y.Z-1.x86_64.rpmThe package installs the four binaries (rustbgpd, rbgp,
rs-config-render, and birdwatcher-adapter) to /usr/bin, the hardened
systemd singleton and opt-in per-handle template units (see
systemd below — ExecStart pointed at /usr/bin), man pages and
shell completions, and creates
the rustbgpd system user (imperatively at install time, plus a
declarative sysusers.d file). A starter config lands at
/etc/rustbgpd/config.toml (mode 0640 root:rustbgpd, never
overwritten on upgrade) — the minimal profile
with its state paths set to /var/lib/rustbgpd. Version-matched monitoring
payloads land under /usr/share/doc/rustbgpd/monitoring/. Then:
sudo $EDITOR /etc/rustbgpd/config.toml # set ASN, router_id, neighbors
sudo systemctl enable --now rustbgpdThe kernel-dataplane capability drop-in ships as documentation at
/usr/share/doc/rustbgpd/examples/rustbgpd-dataplane.conf; copy it
into /etc/systemd/system/rustbgpd.service.d/ only for deployments
that program the kernel (see below). Config-mutating RPCs (rbgp neighbor add, gNMI Set) need /etc/rustbgpd writable by the
service user; the packaged default (0750 root:rustbgpd, read-only
for the daemon) supports the SIGHUP-reload workflow — chown rustbgpd /etc/rustbgpd /etc/rustbgpd/config.toml if you want runtime
mutation.
Removing the package (apt-get remove/purge, dnf remove) stops and
disables rustbgpd and every rustbgpd@ instance before the binaries are
deleted, so the unit's ExecStopPost still releases any EVPN core-isolation
links. The stop is graceful and withdraws the daemon's routes from its peers.
/var/lib/rustbgpd and the rustbgpd user are kept; purge also deletes
/etc/rustbgpd/config.toml on Debian and Ubuntu.
Verifying release artifacts
Each architecture has a checksums-<arch>.txt manifest covering its
tarball and native packages. Select the exact downloaded filename before
passing the row to sha256sum; checking the whole manifest fails when the
other package formats have not also been downloaded:
SUFFIX=linux-amd64
ARTIFACT=rustbgpd-linux-amd64.tar.gz
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/${ARTIFACT}"
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$ARTIFACT" '$2 == file || $2 == "./" file { print }' \
"checksums-${SUFFIX}.txt" | sha256sum -c -This detects a download that differs from the manifest published with the
GitHub release; it does not independently authenticate GitHub. The committed
Cargo.lock is the exact Rust dependency inventory for a tagged source tree.
Release binaries and containers are provided under the repository's license
terms and warranty disclaimers.
Monitoring payloads
Every release archive carries its version-matched Grafana dashboards,
Prometheus alert rules, and promtool test suite under share/monitoring/.
Native packages put the same files under
/usr/share/doc/rustbgpd/monitoring/. Keep rustbgpd-alerts.yml beside
rustbgpd-alerts_test.yml: the suite deliberately uses the relative
rule_files entry rustbgpd-alerts.yml.
Validate the shipped rules before loading them:
# From an extracted release tarball:
(cd share/monitoring && promtool test rules rustbgpd-alerts_test.yml)
# Or from a native package:
(cd /usr/share/doc/rustbgpd/monitoring && \
promtool test rules rustbgpd-alerts_test.yml)Copy or reference rustbgpd-alerts.yml from Prometheus's rule_files
configuration, then import rustbgpd-overview.json and, when operating EVPN,
the Alpha rustbgpd-evpn.json dashboard in Grafana. The Grafana
guide supplies the matching scrape job and dashboard setup; keep
the payloads from one rustbgpd release together so rules and metrics do not
drift across versions.
From source
# Prerequisites: Rust ≥ 1.95, protobuf-compiler
sudo apt-get install -y protobuf-compiler # Debian/Ubuntu
git clone https://github.com/lance0/rustbgpd
cd rustbgpd
# Builds exactly what a deployment ships: daemon, rbgp, the route-server
# config renderer, and Birdwatcher adapter. (`--workspace` would also build
# dev/bench helpers you don't need.)
cargo build --release -p rustbgpd -p rustbgpctl -p rs-config-render -p birdwatcher-adapter
sudo install -m 0755 \
target/release/{rustbgpd,rbgp,rs-config-render,birdwatcher-adapter} \
/usr/local/bin/Container image
The current release workflow can build and runtime-verify images natively on
Linux amd64 and arm64, then publish a single multi-arch (amd64+arm64) manifest
on a tagged release. This does not backfill existing images: :0.68.0 and
:0.68 remain amd64-only. :latest follows the newest non-prerelease release
and inherits that release's platform set. The workflow configuration produces
three tag flavors with docker/metadata-action in
.github/workflows/container.yml:
| Tag | Resolves to | Updates on |
|---|---|---|
:X.Y.Z | exact version and its published platform set | nothing (immutable) |
:X.Y | latest patch in the X.Y minor | each X.Y.Z release |
:latest | latest non-prerelease release, including its platform set | each minor or patch release |
Major-minor is the usual operator default — auto-receives bug-fix
releases but pins against minor-version churn. The examples in this
document use :latest so the tag stays copy-pasteable across releases.
Substitute the major-minor tag of the series you standardize on:
docker pull ghcr.io/lance0/rustbgpd:latestThe published image is the Dockerfile's default runtime target: a lean daemon
with rbgp and birdwatcher-adapter, running as a nonroot rustbgpd user. No
dev/test/bench helpers are included. rs-config-render is deliberately left
out too: the IXP refresh loop is a host-side cron job that runs
arouteserver and the renderer next to each other and hands the daemon a
finished config directory, so the renderer belongs on the host, not in the
daemon's image. Install it from the tarball. If you'd rather build the
image locally:
docker build -t rustbgpd:latest .The development image — adds evpn-tester / evpn-monitor,
iproute2, and the interop start script, and runs as root for lab
plumbing — is the dev target. It's what the M-series interop suite
and the soak harnesses run against:
docker build --target dev -t rustbgpd:dev .Birdwatcher adapter
Native packages install birdwatcher-adapter at
/usr/bin/birdwatcher-adapter; the production container installs it at
/usr/local/bin/birdwatcher-adapter. For a same-host deployment, point it at
the daemon's absolute Unix socket and run it under a supervisor as the
rustbgpd user:
sudo -u rustbgpd birdwatcher-adapter \
--grpc-addr unix:///var/lib/rustbgpd/grpc.sock \
--listen 127.0.0.1:8080TCP remains supported with --grpc-addr http://127.0.0.1:50051. The adapter
may start before the daemon/socket: REST requests return 502 Bad Gateway
while gRPC is unavailable and recover without restart. No service unit is
installed; use the supervisor and lifecycle policy appropriate for
the Alice-LG deployment. See the
adapter README for bearer-token
and endpoint details. This convenient owner-only socket authenticates as the
operator-tier local-operator; use a dedicated token-authenticated listener
mapped to observer when least privilege is required.
systemd
A hardened unit lives at
examples/systemd/rustbgpd.service.
It runs the daemon unprivileged by default: a dedicated rustbgpd
system user, the standard sandbox set (NoNewPrivileges,
ProtectSystem=strict, ProtectHome, PrivateTmp, StateDirectory /
RuntimeDirectory), and a capability set of exactly
CAP_NET_BIND_SERVICE (bind port 179 as non-root). Deployments with
interface-only neighbors
also need the opt-in CAP_NET_RAW drop-in.
Releases also ship the opt-in
rustbgpd@.service for IXP Manager.
Literal %i is the exact router handle: it selects
/var/lib/rustbgpd/%i/activation/current/config.toml, a private per-handle
runtime/UDS, and no writable access to the shared lifecycle fence. Pre-create
each exact handle and the one shared fence directory before the first lifecycle
run; authorize and invoke only literal instances, never a wildcard sudoers rule:
handle=rs1-ipv4
sudo install -d -m 0700 -o rustbgpd -g rustbgpd \
"/var/lib/rustbgpd/$handle" "/var/lib/rustbgpd/$handle/activation" \
/var/lib/rustbgpd/ixp-manager-host
sudo systemctl enable "rustbgpd@$handle"Use distinct runtime/activation/UDS paths for every handle and the same
/var/lib/rustbgpd/ixp-manager-host for all lifecycle commands on that host.
Notes on the sandbox:
-
CAP_NET_BIND_SERVICElets the daemon bind port 179 without running as root. Nothing else is granted by default — in particular notCAP_NET_ADMIN, so the unprivileged unit cannot program the kernel. -
ProtectSystem=strict+ explicitReadWritePathsconfines the daemon to/var/lib/rustbgpd(state) and/etc/rustbgpd(config). -
ExecStartpasses--runtime-state-dir /var/lib/rustbgpd; the daemon refuses to start when that differs from the config'sruntime_state_dir, so change both together.ExecStopPost=runsrustbgpd --release-core-isolation --runtime-state-dir /var/lib/rustbgpdafter every stop, crash or failed start: it raises CE-facing links that an EVPN core-isolation hold left down, needsCAP_NET_ADMINonly when there is something to release, and exits 0 when there is nothing to release. -
ExecReload=kill -HUPis the supported reload path.systemctl reloadreturns once the signal is sent; automation that needs the outcome runsrbgp reload, which sends the same signal, waits for the reload to settle and exits non-zero unless it completed. See the reload matrix for which fields hot-apply vs. need a restart. -
Type=notifyis load-bearing. The daemon sendsREADY=1only after every configured gRPC listener is bound, the configured peer roster is installed, and BGP ingress is active — the same boundary/readyzcannot go green before — sosystemctl startand units orderedAfter=rustbgpd.servicewait for a daemon that can accept sessions, not for a forked process. A gRPC bind failure prevents peer registration, BGP ingress, telemetry, dial-out, and readiness publication before entering shortened coordinated teardown.STOPPING=1marks the start of coordinated shutdown.TimeoutStartSec=10minbounds initial roster installation beforeREADY=1. -
WatchdogSec=5minrestarts a daemon whose core actors have stopped answering. Every 10 seconds the daemon asks the PeerManager and the RIB actor for an answer, waits up to 5 seconds, and sendsWATCHDOG=1when both answered. This is a liveness sample, not/readyz:- A RIB that answers "stalled" from inside a long policy transition,
selection-deferral release or peer teardown is working, not dead.
/readyzreturns503for it; the watchdog keeps pinging. - Only silence withholds the ping: an actor that does not answer within
5 seconds, a closed channel or a dropped reply. A ping that the notify
socket refuses counts the same way. The daemon logs a WARN, increments
bgp_systemd_watchdog_pings_skipped_totaland samples again one second later, so a silent actor is asked every 6 seconds (5 seconds of waiting, then the 1-second retry). - The sampler runs on its own thread. It does not depend on the telemetry listener being configured, and a busy daemon worker cannot delay it.
- The sample ignores the shutdown gate, so pings continue through
coordinated shutdown for as long as the actors answer. Once the peer
manager starts draining sessions it stops answering, so the counter
moves until the process exits. That cannot abort a stop: systemd stops
enforcing the watchdog when it receives
STOPPING=1, andTimeoutStopSecbounds the stop from then on.
The watchdog does not bound how long the RIB may keep answering "stalled". A unit of work that keeps making progress is never aborted, however long it runs. Alert on it instead: the daemon samples the RIB every 5 seconds and exports
bgp_rib_stalled(1 while the latest sample was a stalled answer). The shippedBgpReadinessRibStalledrule fires when it reads 1 on every evaluation for ten minutes. It needs only a/metricsscrape, no/readyzprobe. Restart by hand only if the work is not finishing.PID 1 independently sends SIGABRT five minutes after the last ping, and
Restart=on-failurecovers that expiry. Measured from the moment an actor stops answering, that is about five minutes: 290 to 300 seconds when answers were prompt, and 285 to 305 seconds at the edges. A sample that took almost 5 seconds to answer makes the last ping up to 15 seconds old, and an actor that stops while the other's answer to the same sample is still on its way lets one more ping out up to 5 seconds later. A stopped process (SIGSTOP, a debugger) is aborted 285 to 300 seconds after it stops. With anotherWatchdogSecthe sample period is a quarter of it and the sample deadline an eighth, capped at 10 and 5 seconds, so the abort comes between five eighths and nine eighths of the timeout after an actor stops. - A RIB that answers "stalled" from inside a long policy transition,
selection-deferral release or peer teardown is working, not dead.
-
TimeoutAbortSec=2minbounds the wait after that SIGABRT. The daemon does not handle the signal, so the kernel writes a core dump and the restart waits for it; every session is already down. Two minutes is enough for a multi-gigabyte core, and systemd truncates a larger one with SIGKILL. Without the setting systemd would waitTimeoutStopSec, 32 minutes. Raise it if complete cores of a very large table matter more than restart time. It needs systemd 243 or newer; older versions ignore it with a warning.Without
NOTIFY_SOCKET— a foreground run or Docker — every notification is a silent no-op. A native unit opting out must disable all inherited notify behavior:[Service] Type=simple NotifyAccess=none WatchdogSec=0The container unit keeps
Type=simplebecause Docker, not the daemon, is its main process. -
Restart=on-failureis load-bearing. The daemon exits0only on an operator-initiated shutdown (SIGINT/SIGTERM, theShutdownRPC) and1on a component failure it cannot recover from in place. Startup exits immediately if legacy BGP mode binds neither family, explicitlisten_addressescannot bind every configured endpoint, or a configuredprometheus_addrhealth listener cannot bind, or any configured gRPC listener cannot bind. An unexpected RIB manager, peer manager, RPKI subsystem task, BGP listener task, inbound accept-forwarding task, or injection lease expiry task exit instead runs the coordinated peer teardown before exit 1. These components are not respawned in place; the supervisor is the recovery path. The gRPC server and the metrics/readiness server are the exception: the daemon restarts them in place and keeps BGP running (see gRPC server dies unexpectedly and Metrics/readiness server exits unexpectedly). Exit 70 is also a failure: it is the runtime-config settlement watchdog's fail-stop recovery request and must remain restartable.RestartSec=5retries a transient failure, whileStartLimitBurst=5andStartLimitIntervalSec=10minstop a deterministic failure from flapping forever. After correcting config authority or permissions, recover a rate-limited unit withsystemctl reset-failed rustbgpdfollowed bysystemctl start rustbgpd. An explicitsystemctl stopsuppresses restart, andTimeoutStopSec=32mingives an already-owned mutation longer than its 30-minute watchdog plus five-second terminal grace to settle or fail-stop. Nothing else in the stop path needs that long: waits outside settlement ownership are bounded in seconds, and a second SIGTERM or SIGINT skips the ones that have no deadline, soTimeoutStopSecdoes not need to change. A second signal while a mutation is still settling instead forces the watchdog's fail-stop at once (fence_reason="operator_forced", exit 70 after the grace). The supervisor consequence depends on how the stop was requested: raw signals to a running unit end in exit 70, an unclean exit code thatRestart=on-failurerestarts, while a unit stopped explicitly withsystemctl stopis never restarted automatically whatever its exit status. Recovery from the persisted transaction runs on the next actual start either way.rbgp doctorreports listener failures through itsbgp.listenercheck.
Installation
The Debian/RPM packages perform all of the
below on install (unit at /lib/systemd/system, ExecStart at
/usr/bin). The tarball commands above use its share/systemd/ tree.
The following paths are only for a source checkout:
sudo useradd --system --home-dir /var/lib/rustbgpd \
--shell /usr/sbin/nologin rustbgpd
sudo install -m 0644 examples/systemd/rustbgpd.service \
/etc/systemd/system/rustbgpd.service
sudo install -d -m 0755 /etc/rustbgpd
sudo install -m 0640 examples/minimal/config.toml /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml # readable by the service user
$EDITOR /etc/rustbgpd/config.toml # set ASN, router_id, neighbors
sudo systemctl daemon-reload
sudo systemctl enable --now rustbgpdInterface-only neighbors: opt-in privilege drop-in
When at least one neighbor has interface without address, grant
CAP_NET_RAW for ICMPv6 discovery, including with
[global.unnumbered] router_advertisements = false. Static link-local
neighbors with an explicit address and RS/RR deployments without
interface-only neighbors do not need this capability. The base unit keeps its
existing capability set; install the separate example from the source checkout:
sudo mkdir -p /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 examples/systemd/rustbgpd-unnumbered.conf \
/etc/systemd/system/rustbgpd.service.d/rustbgpd-unnumbered.conf
sudo systemctl daemon-reload
sudo systemctl restart rustbgpdThe drop-in adds CAP_NET_RAW to AmbientCapabilities and
CapabilityBoundingSet. Install the dataplane drop-in alongside it when the
configuration also programs the kernel. For a per-handle unit, place the
capability drop-in under the exact rustbgpd@<handle>.service.d/ directory.
A peer RA with a non-zero Router Lifetime can install a kernel default route
independently of BGP if the host accepts RA defaults. Disable those defaults
on dedicated peering ports (for example, net.ipv6.conf.<port>.accept_ra=0)
when routing must follow BGP. rustbgpd receives discovery packets on its raw
socket regardless of that sysctl. Its own RAs also carry a non-zero Router
Lifetime (three times ra_interval_secs, 30 seconds by default), so neighboring
hosts must apply the same RA-default policy when their routing should follow BGP.
See the interface-only configuration.
Kernel dataplane: opt-in privilege drop-in
If — and only if — the config programs the kernel ([[fib_tables]]
per ADR-0061, install_blackhole_discard, or the EVPN VTEP/IRB
dataplane), the daemon needs CAP_NET_ADMIN for rtnetlink route /
nexthop programming, VXLAN + bridge FDB writes, and the
RTNLGRP_NEIGH subscription behind local-MAC learning. Grant it via
the shipped drop-in
examples/systemd/rustbgpd-dataplane.conf
instead of editing the base unit — systemd merges capability sets, so
the drop-in adds CAP_NET_ADMIN on top of the unprivileged default:
sudo mkdir -p /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 examples/systemd/rustbgpd-dataplane.conf \
/etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.conf
sudo systemctl daemon-reload
sudo systemctl restart rustbgpdRR-only boxes should not install this drop-in — the unprivileged default is the point.
Dataplane host caveat: if rustbgpd programs the kernel on a host
that also runs systemd-networkd, set ManageForeignRoutes=no,
ManageForeignRoutingPolicyRules=no, and (where supported)
ManageForeignNextHops=no in networkd.conf. The defaults are
yes, and networkd will delete routes and next-hop groups it did
not create — including rustbgpd's, especially across a daemon
restart (ADR-0079). Co-residency with another daemon that claims
proto bgp kernel state (e.g. FRR zebra) is unsupported for the
same reason.
Reserved nexthop-ID ranges (EVPN dataplane)
When the EVPN VTEP/IRB dataplane is enabled, rustbgpd exclusively
owns four kernel nexthop-ID ranges, distinguished by the ID's high
nibble (deliberately offset from FRR's 0x1000/0x2000 tags so both
daemons can't collide on NLM_F_REPLACE):
| Range | Use |
|---|---|
0x3000_0000–0x3FFF_FFFF | L2 per-VTEP FDB nexthop members (ADR-0059 aliasing ECMP) |
0x4000_0000–0x4FFF_FFFF | L2 FDB nexthop groups (ADR-0059) |
0x5000_0000–0x5FFF_FFFF | L3VXLAN per-VTEP nexthop members (all-active Type 5) |
0x6000_0000–0x6FFF_FFFF | L3VXLAN FDB nexthop groups (all-active Type 5) |
This is a deployment contract: no co-resident netlink writer may
create nexthop objects with IDs in these ranges (ip nexthop add id …, another routing daemon, orchestration tooling). rustbgpd
adopts objects it finds there across restarts and garbage-collects
the ones nothing references — an unrelated agent's object parked in
the range would be treated as rustbgpd state.
rustbgpd validates the shape of every in-range object it dumps at
startup adoption and during drift recovery (FDB flag, member vs
group structure matching the tag). An in-range object that does not
look like anything rustbgpd writes fails closed per ID: the object is
left untouched, excluded from adoption and from every reap/delete
path, its ID is quarantined so the allocator can never hand it out
(and thus never NLM_F_REPLACE over it), and the violation is
logged once and counted on the
evpn_foreign_nhid_range_conflicts_total Prometheus counter. A
nonzero counter means a co-resident writer is violating this
contract — find and remove that writer; the quarantined IDs are not
reclaimed until a daemon restart after the foreign objects are gone.
Note this shape check cannot distinguish a foreign object that is
byte-for-byte identical to what rustbgpd writes; stamping
nh_protocol on owned nexthop objects as a provenance marker is a
possible future defense-in-depth, not a substitute for this
contract (any netlink writer can set the same protocol value — it is
provenance, not kernel-enforced exclusivity).
Operational checks
systemctl status rustbgpd # is it running?
journalctl -u rustbgpd -f # follow logs
systemctl reload rustbgpd # SIGHUP → config reload
sudo rbgp reload # SIGHUP, wait, print the outcome receipt
rbgp config last-reload # outcome of the latest settled reload
rbgp config diff /etc/rustbgpd/config.toml # installed file vs running config
systemctl restart rustbgpd # full restart (state drained)Docker / Docker Compose
A worked starting point at
examples/docker-compose/
brings up rustbgpd alongside an FRR peer that advertises sample
prefixes — no real routers required.
cd examples/docker-compose
docker compose up -d --build
docker compose exec rustbgpd \
rbgp -s http://127.0.0.1:50051 neighbor
docker compose downFor your own deployment:
-
Image name and command: published GHCR version tags have no leading
v: useghcr.io/lance0/rustbgpd:0.83.0, notghcr.io/lance0/rustbgpd:v0.83.0. The older:0.68.0image is amd64-only; it is not backfilled when a later release publishes multiple platforms. The production image runs as uid/gid 999 — pinned in theDockerfileruntime stage (useradd --uid 999 --gid 999andUSER 999:999), not allocated by the base image, and asserted against the built image by the container workflow — and its default command is exactlyrustbgpd /etc/rustbgpd/config.toml. If the config is mounted under another container filename, pass that filename explicitly after the image, for examplerustbgpd /etc/rustbgpd/router.toml. -
Bridge networking and state: mount a volume at the daemon's
runtime_state_dirso the GR restart marker and FIB and BLACKHOLE ownership receipts survive container restarts. The daemon runs as uid/gid 999 and rewrites its config in place, so both host directories must be owned by that uid and the config must already exist — the image's default command isrustbgpd /etc/rustbgpd/config.tomland it will not create one:# One-time host prep. sudo install -d -o 999 -g 999 /etc/rustbgpd /var/lib/rustbgpd docker run --rm ghcr.io/lance0/rustbgpd:latest \ rustbgpd --init-config edge > config.toml # Edit config.toml for your ASN, router ID, peers, and policy. sudo install -o 999 -g 999 -m 0600 config.toml /etc/rustbgpd/config.toml docker run --rm -d \ --name rustbgpd \ --network=bridge \ --stop-timeout=1920 \ -v /etc/rustbgpd:/etc/rustbgpd \ -v /var/lib/rustbgpd:/var/lib/rustbgpd \ -p 179:179 \ -p 9179:9179 \ --ulimit nofile=65536:524288 \ ghcr.io/lance0/rustbgpd:latestThis recipe succeeds because Docker sets
net.ipv4.ip_unprivileged_port_start=0inside its default bridge network namespace. Port 179 is therefore not privileged there; no capability flag is responsible for the bind. AddNET_ADMINonly when the container actually programs its network namespace. The bridge path is the supported choice when the daemon does not need to bind a particular host/peering-LAN address.The
edgeprofile is used because itsruntime_state_dir,grpc_udspath, and0.0.0.0metrics bind are already the container forms; thelabprofile's/tmpstate directory and127.0.0.1metrics bind both need editing first.
Host networking and privileged BGP ports
Host networking is the second supported container path, and is required when
listen_addresses must name a host/peering-LAN address, a multihop session
pins its source, or the daemon must program the host network namespace. It does
not inherit Docker's bridge-netns low-port setting:
| Docker network mode | Effective net.ipv4.ip_unprivileged_port_start | Image uid 999 can bind :179 |
|---|---|---|
| default bridge | 0 in the container namespace | yes |
--network=host | the host value (commonly 1024) | no when the host value is above 179 |
--cap-add=NET_BIND_SERVICE is not sufficient for the image's uid 999 in
host mode. Docker does not provide that capability as an ambient capability to
the non-root image process, so it is unavailable after the uid 999 exec. This
differs from the bare-binary systemd unit above, whose
AmbientCapabilities=CAP_NET_BIND_SERVICE is effective and remains correct.
Use exactly one of these deployment paths:
-
Keep the default bridge network and published ports when address pinning is unnecessary; the image continues to run as uid 999.
-
Use
--network=hostwith--user=rootand bind a root-owned host state directory at/var/lib/rustbgpd:sudo install -d -o root -g root -m 0755 /etc/rustbgpd sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml docker run --rm -d \ --name rustbgpd \ --network=host \ --user=root \ --cap-drop=ALL \ --cap-add=NET_BIND_SERVICE \ --stop-timeout=1920 \ --ulimit nofile=65536:524288 \ --mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \ --mount=type=bind,source=/var/lib/rustbgpd,target=/var/lib/rustbgpd \ ghcr.io/lance0/rustbgpd:0.83.0
Do not omit the host state bind when overriding the image user. The directory
baked into the image is owned by uid 999; a root daemon expects uid 0. The
runtime-state owner guard treats that mismatch as unsafe and refuses to start
with refusing to start: runtime_state_dir must be a directory owned by the daemon user and not group- or world-writable. v0.69.0 and later images
also reject the default gRPC UDS listener at startup. A root-owned,
non-group/world-writable bind mount makes the authority match the running uid. The socket parent must allow the daemon
read/write/search access, with no symlinks or untrusted writable ancestors;
see Unix socket path integrity.
Adding NET_BIND_SERVICE to uid 999 is not a third supported host-network
recipe. If changing the host's low-port sysctl is part of a separately managed
host policy, validate that policy outside this unit; the shipped container
guidance does not mutate the host sysctl.
The root host-network path drops Docker's entire default capability set, then
adds back only NET_BIND_SERVICE for port 179. Root makes that explicitly
granted capability usable; applying the same add-cap flag to uid 999 remains
ineffective because Docker supplies no ambient capability to that non-root
exec.
The command above and the systemd unit below are control-plane defaults. A
configuration that programs the host FIB or interfaces also needs
--cap-add=NET_ADMIN on the same root host-network path, as described in
kernel dataplane privileges; that
opt-in does not replace the root-user and root-owned-state requirements.
Interface-only neighbor discovery also needs --cap-add=NET_RAW on that root
host-network path, even when router_advertisements = false. The flag does
not supply an ambient capability to the image's default non-root process.
Static link-local neighbors with an explicit address do not need NET_RAW.
systemd-supervised host-network container
examples/systemd/rustbgpd-container.service
generalizes the host-network recipe above. It keeps docker run attached so
the daemon's exit status drives Restart=on-failure, maps systemctl reload
to SIGHUP, gives Docker the full 32-minute settlement grace and systemd a
33-minute outer margin, and makes the registry pull best-effort so an
explicitly selected cached image remains usable during a registry outage. A
missing or empty RUSTBGPD_IMAGE fails closed before any Docker command runs.
Install the unit and its required image environment file after creating the root-owned state directory:
sudo install -d -o root -g root -m 0755 /etc/rustbgpd
sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd
sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml
sudo install -m 0644 examples/systemd/rustbgpd-container.service \
/etc/systemd/system/rustbgpd-container.service
sudo tee /etc/rustbgpd/rustbgpd-container.env >/dev/null <<'EOF'
RUSTBGPD_IMAGE=ghcr.io/lance0/rustbgpd:0.83.0
EOF
sudo chmod 0644 /etc/rustbgpd/rustbgpd-container.env
sudo systemctl daemon-reloadThe overridable host defaults are RUSTBGPD_CONFIG_DIR=/etc/rustbgpd and
RUSTBGPD_STATE_DIR=/var/lib/rustbgpd; both are bind-mounted at their standard
container paths. RUSTBGPD_CONFIG_FILE=/etc/rustbgpd/config.toml selects the
file inside the read-only config mount. Put any overrides beside
RUSTBGPD_IMAGE in the environment file. An upgrade is one image-tag edit
followed by systemctl restart rustbgpd-container; version tags still omit the
leading v. RUSTBGPD_IMAGE is required and must be non-empty; the unit
checks it before its best-effort pull.
The unit deliberately mounts the config directory read-only. Runtime mutation
(rbgp neighbor add, policy edits, gNMI Set, and rbgp config apply) is
therefore excluded: those calls cannot persist and are rejected. Supporting
them requires an explicit change to the mount and root-owned config-directory
write contract; the shipped unit is for externally managed config plus SIGHUP.
Before the first start, and before restarting after an image or config change, validate the exact externally managed config directory and selected image without touching the running container:
docker run --rm \
--user=root \
--cap-drop=ALL \
--mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \
ghcr.io/lance0/rustbgpd:0.83.0 \
rustbgpd --check --strict /etc/rustbgpd/config.tomlOnly enable the service after that check succeeds:
sudo systemctl enable --now rustbgpd-containerThe image's declared healthcheck remains active under systemd supervision and
runs rbgp health --liveness (gRPC handler liveness, not core-actor readiness)
against the default state-directory UDS. An
unhealthy result remains observable with
docker inspect --format '{{.State.Health.Status}}' rustbgpd, but Docker health
status does not terminate the attached process and therefore does not drive
systemd's Restart=on-failure. If the config moves the socket, set
RUSTBGPD_ADDR for the container by extending the unit's docker run command.
-
File descriptors:
--ulimit nofilesets the ceiling. rustbgpd raises its soft limit to the hard limit at startup, so the hard limit must clear the 4096 floorrbgp doctorenforces; peers exhaust descriptors at scale. -
Writable config directory: mount
/etc/rustbgpdread-write and make it owned by the daemon user (chownthe host directory to the container'srustbgpduid) if you use runtime mutation —rbgp neighbor add, policy edits, gNMISet,rbgp config apply. Config persistence rewrites the file with a temp-file + rename, so the directory, not just the file, has to be writable, and every mutating RPC is rejected without it. Mount it:roonly when the config is managed entirely from outside and reloaded with SIGHUP. -
Settlement fail-stop:
--stop-timeout=1920gives the daemon a 32-minute Docker stop grace; the shipped container unit'sTimeoutStopSec=33minis the supervisor margin around that operation. A failure to create the initial persistence stage rejects cleanly before runtime mutation. After runtime mutation begins, a failed rename isNotPublished: only complete acknowledged compensation returns cleanly; otherwise known divergence fences recovery. A post-rename directory-fsync failure isPublicationAmbiguous: the complete visible candidate is adopted without rollback, and restart selects durable authority. A read-only bind-mounted config file can have a writable parent yet still fail replacement asNotPublished. A fenced daemon marks readiness unavailable, closes persisted-mutation admission, and exits 70 within five seconds; a silent owner is fenced at 30 minutes and exits five seconds later. This standalone command has no restart policy; use bounded supervisor retries and inspect durable config authority before recovery. -
Health: the image runs
rbgp health --livenessevery 30 seconds. Success means only that the authenticated gRPC handler answered; it does not check core-actor readiness, BGP convergence, or forwarding. This is an intentional change from the previous image probe, which usedGetHealthand checked core actors. To retain that behavior, start the container with--health-cmd='rbgp health'. Ordinaryrbgp healthkeeps its detailed output and readiness behavior. The liveness RPC requires only thereadlistener tier; credentials and principal-role checks still apply. Failed, refused, or unreachable probes exit non-zero. The default endpoint isunix:///var/lib/rustbgpd/grpc.sock; setRUSTBGPD_ADDRon the container if configuration moves it. HTTP/livezis also available on the metrics listener;/readyzretains core-actor readiness checks. -
Logs:
[global.telemetry] log_format = "json"emits structured JSON. Bound retention when creating the container, for example with--log-driver=json-file --log-opt max-size=10m --log-opt max-file=3ondocker run. Add these flags to the container systemd unit'sdocker runcommand too. For Compose, setlogging.driver: json-fileandlogging.options: {max-size: "10m", max-file: "3"}on the service. These limits keep approximately 30 MB per container; recreate existing containers to apply them. The image cannot configure the host logging driver. Forward logs to an aggregator when longer retention is needed. Successfulreadauthorization audits are DEBUG; counters and existing denied, error, sensitive-read, and mutation audit levels are unchanged. -
Networking: Linux FIB integration and BFD require access to the network namespace they operate on. For host addresses, use the root host-network path above; if the configuration also programs the host FIB or interfaces, add its documented
NET_ADMINopt-in. A bridge container affects only its own namespace. The interop suite runs rustbgpd as a containerlabkind: linuxnode, which is the cleanest reference setup for an isolated lab namespace.
Kubernetes probes
Kubernetes does not run the image's Docker health check; configure
probes
against the telemetry HTTP listener instead. The probes exist only when
prometheus_addr is set under [global.telemetry] (the edge profile binds
0.0.0.0:9179). Their status codes and bodies are listed under
HTTP probes.
containers:
- name: rustbgpd
image: ghcr.io/lance0/rustbgpd:latest # pin a release tag in production
ports:
- name: telemetry
containerPort: 9179
startupProbe:
httpGet: {path: /livez, port: telemetry}
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 60 # 10 minutes, the unit's TimeoutStartSec
readinessProbe:
httpGet: {path: /readyz, port: telemetry}
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
livenessProbe:
httpGet: {path: /livez, port: telemetry}
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
terminationGracePeriodSeconds: 90 # liveness kills only; Kubernetes 1.25+
terminationGracePeriodSeconds: 1920 # ordinary stops; same as docker --stop-timeoutReadiness uses /readyz with the Kubernetes defaults: a 10 s period, a
1 s timeout and three failures. /readyz answers 200 ready when the
PeerManager and RIB actors both respond within one 200 ms deadline and the
gRPC server is not in the middle of an in-place restart. It means
the control plane is up and admitting work. It does not mean the daemon has
converged, received any routes, or established any peers; a daemon with zero
Established peers can be ready.
That is deliberate for a route server or reflector. Peers open BGP sessions
to a configured speaker address, not through a load-balanced Service, so
readiness does not steer them toward a healthy replica. If readiness
waited for Established peers, an outage on the peer side would mark every
rustbgpd pod unready at once: the failure would appear to be in rustbgpd,
rolling updates would stall, and restarting the pods could not fix it. Watch
session state through its own signals instead, such as the
BgpSessionNotEstablished rule in
examples/prometheus/rustbgpd-alerts.yml.
Liveness uses /livez. It returns 200 ok whenever the HTTP listener
accepts a connection and checks no actors. The listener has its own thread,
so /livez keeps answering while the daemon's runtime workers are saturated.
It restarts a process that is gone or cannot serve HTTP at all. It does not
restart a daemon whose RIB or PeerManager has deadlocked while the listener
still answers; Kubernetes has no equivalent of the systemd watchdog's
actor check (see systemd). Alert on that case instead: a
/readyz that stays 503, or
bgp_readiness_probe_failures_total{reason=~"(peer_manager|rib)_(timeout|unavailable)"}
still rising after several minutes, means the actors are not answering.
The daemon restarts the telemetry listener's HTTP server in place if that
server crashes, instead of exiting (see
Metrics/readiness server exits unexpectedly).
A restart gap is at most 10 seconds, under the three consecutive failures
this probe needs, so a server that comes back does not get the pod killed.
A server that can never restart still fails liveness, and Kubernetes then
restarts a daemon whose BGP side was healthy. To let BGP outlive that, point
liveness at something other than this listener or raise failureThreshold.
The gRPC server is restarted in place the same way. While it is down
/readyz answers 503 not ready: gRPC server is restarting, so the pod is
unready for the length of the restart; /livez is not affected.
Do not point liveness at /readyz, with any failureThreshold. /readyz
fails for conditions that are not a dead daemon:
- The RIB reports an export-policy transition, selection-deferral release or peer teardown as stalled once it has run for 30 seconds, and keeps reporting it until the work finishes. The actor is answering and making progress the whole time. On a large table that work can outlast any threshold chosen in advance; a restart then drops every session and the daemon repeats the same work after it comes back.
- Either core actor can miss the 200 ms deadline under heavy load and recover on its own.
The Kubernetes documentation warns that liveness probes that fail under load cause restarts that add to the load and can cascade. A restart of a BGP speaker resets every session it holds, so a liveness probe must fail only when a restart is the fix.
Startup. The listener is bound early, but nothing answers HTTP, not even
/livez, until startup wiring has finished and the configured peer roster is
installed, the same boundary as systemd READY=1. Until then, probes connect
and time out. The startup probe uses /livez for that reason: its first
200 already means startup finished, and unlike /readyz it cannot be held
red by a long RIB unit of work during initial table loading. It holds
liveness off for the 10 minutes the
systemd unit allows (TimeoutStartSec=10min). BGP ingress opens at that
boundary, so routes from peers, including full tables, load while the probes
already answer; the startup probe does not need to cover table loading.
Grace periods. Set the pod-level terminationGracePeriodSeconds to at
least the stop budget described under systemd: an owned
runtime-config mutation can take up to its 30-minute settlement watchdog plus
a five-second grace to finish. During coordinated shutdown /readyz returns
503 not ready: daemon is shutting down.
A failed liveness probe only starts that same termination. Without a probe-level override, Kubernetes applies the pod-level grace period to a liveness kill too, so the time to restart a daemon that stopped serving HTTP is the detection time plus up to 32 minutes of shutdown. A wedged daemon usually cannot shut down cleanly: coordinated shutdown waits without a deadline for the PeerManager to drain its peers, and only a second termination signal skips that wait. Kubernetes sends one SIGTERM, then SIGKILL when the grace period ends.
The snippet therefore sets a
probe-level terminationGracePeriodSeconds
of 90 seconds on the liveness probe. The Kubernetes documentation supports
the field from 1.25 and marks it stable since 1.28. The shutdown stages that
have deadlines add up to about 70 seconds: a five-second runtime-config
drain, then the
pre-Cease ceilings of 30 seconds for the warm
checkpoint, 5 seconds for the restart marker, 10 seconds for the EVPN
runtime-apply fence and 5 seconds for each of four EVPN actor drains. 90
seconds covers them with headroom if the actors recover.
The 90 seconds is an intentional cap, not an estimate of a clean stop. A daemon that is still wedged when it expires is killed, and the PeerManager drain, which has no deadline, is deliberately not waited for. Graceful drain is moot by then: the process could not answer a plain HTTP request. The systemd path is no gentler: the shipped unit keeps systemd's default watchdog signal, SIGABRT, which the daemon does not handle, so a watchdog expiry ends the process without coordinated shutdown. In either case a runtime-config mutation that was cut short is reverted or adopted from its durable state on the next start (see settlement-watchdog.md). On clusters older than 1.25, the pod-level value applies to liveness kills as well.
Containerlab quick start
Containerlab is the easiest way to get a
working rustbgpd ↔ FRR session on your laptop with no real routers.
The simplest topology in the repo is
tests/interop/m0-frr.clab.yml
(abridged below; the file adds the config and start-script binds and the
interface addresses):
name: m0-frr
topology:
nodes:
rustbgpd:
kind: linux
image: rustbgpd:dev
cmd: sleep infinity
frr:
kind: linux
image: quay.io/frrouting/frr:10.7.1
links:
- endpoints: ["rustbgpd:eth1", "frr:eth1"]Bring it up:
# Build the dev image
docker build --target dev -t rustbgpd:dev .
# Deploy
sudo containerlab deploy -t tests/interop/m0-frr.clab.yml
# Start rustbgpd (the node runs `sleep infinity` until you do)
sudo docker exec -d clab-m0-frr-rustbgpd /usr/local/bin/start-rustbgpd.sh
# Check the session from the FRR side
sudo docker exec clab-m0-frr-frr vtysh -c "show bgp summary"
# More checks: the test driver scripts under tests/interop/scripts/ show
# the typical incantations for FRR vtysh + rbgp gRPC.
# Tear down
sudo containerlab destroy -t tests/interop/m0-frr.clab.yml --cleanupFor more topologies (route-reflector, EVPN, FlowSpec, BFD, etc.), see
INTEROP.md.
Recommended first production-ish topology
The smallest deployment that exercises the full operator loop — build → validate → reload → observe — without depending on more than one peer. This is your "did I install this correctly" gate before scaling out.
-
One rustbgpd instance + one FRR peer (or any compliant BGP peer). Provision the peer first; record its address + AS.
-
Prometheus scrape on
:9179. Add the scrape target in your Prometheus config:- job_name: rustbgpd static_configs: - targets: ["10.0.0.1:9179"] -
Config validation. Build the config, then:
rustbgpd --check /etc/rustbgpd/config.tomlErrors are rustc-style with file + line + carets; expect
config OKon success. Run this every time before swapping the live config.A check that finds nothing to flag prints
config OK. A check that flags something still exits 0, but the summary readsconfig VALID, <n> WARNING(S) — NOT a clean checkand the warnings are framed on stderr above it. The warnings identify: a configured eBGP neighbor or dynamic range with no explicit policy in a direction (unfiltered when[global] ebgp_requires_policyis off, carrying no routes in that direction when it is on); an RFC 8212 posture still inherited from legacy omission (rfc8212_secure_default_ready— set an explicit rootconfig_epochplus[global] ebgp_requires_policy); anorr_vantagewith nolinkstate-negotiating neighbor; and, when a gRPC TCP listener sets a positivetls_expiry_warning_seconds, a TLS certificate whose notAfter falls inside that window. None is rejected — a permit-all route server is a legitimate configuration — but none should reach production unnoticed. -
First start.
sudo systemctl start rustbgpd journalctl -u rustbgpd -fWatch for the session to reach
Established:rbgp neighbor -
Edit + reload cycle. Edit a copy of the config, then dry-run the diff of the candidate against the current on-disk config (
--difftakes the candidate; the positional argument is the config to compare against, default/etc/rustbgpd/config.toml):cp /etc/rustbgpd/config.toml /tmp/new-config.toml $EDITOR /tmp/new-config.toml rustbgpd --diff /tmp/new-config.toml /etc/rustbgpd/config.tomlWhen the diff looks right, install the candidate over
/etc/rustbgpd/config.tomlbefore reloading.--diffcalls into the sameConfigDiffmachinery the reload path uses; what it reports is what reload will do. Cross-reference against the reload matrix to confirm which changes hot-apply and which are restart-required. Apply:sudo systemctl reload rustbgpd -
Restart cycle. Confirm the FRR peer reconverges within the GR timer (GR is on by default):
sudo systemctl restart rustbgpd # FRR should retain the routes during the restart window; rustbgpd # advertises R=1 in the GR capability on the next OPEN. -
Observability sanity-check. Read at least one Prometheus counter and one neighbor field:
curl -s http://10.0.0.1:9179/metrics \ | grep -E "bgp_session_established_total|bgp_messages_received_total" rbgp neighbor 10.0.0.2
If all six steps work end-to-end, your install is sound. Scale from there: add more peers, add policy chains, wire BMP / gNMI / MRT according to your needs.
Config validation workflow
These cover the config lifecycle, from bootstrap to reload:
| Command | What it does |
|---|---|
rustbgpd --init-config <lab|edge|route-server> | Print a curated, commented starter TOML to stdout and exit (file output is not yet supported). lab is a minimal single-box profile; edge is an eBGP edge skeleton; route-server is a self-contained IPv4 two-member skeleton with fail-closed import policy and transparent export. Every profile uses a mode-0600 local UDS whose filesystem permissions authenticate access as the implicit local-operator, sets [global] ebgp_requires_policy = true, and passes --check --strict as emitted. Cannot be combined with --check / --diff. |
rustbgpd --check <file> | Parse + validate; print config OK, config VALID, <n> WARNINGS — NOT a clean check (warnings framed on stderr; still exit 0), or a rustc-style diagnostic (exit 1). Does not start the daemon. The running daemon logs the same config advisories as WARN lines: all of them at startup, and after a SIGHUP reload or API config change is adopted, each advisory that the previous config did not raise. |
rustbgpd --check --strict <file> | The same check, but any warning exits 1 instead of 0 — for CI and deployment gates that must not accept a valid-but-risky config. A clean check still exits 0. --strict without --check is an error (exit 2). |
rbgp policy check [--fail-on-empty] [--deny-lints[=LINT,...]] [-j] <file.rpol> | Compile a .rpol file and run its test blocks offline, without a daemon — a CI or editor gate for policy files. Exit 0 = clean, 1 = compile error, 2 = failing test (or no tests under --fail-on-empty), 3 = coverage below a requested threshold, 4 = a lint denied by --deny-lints. -j prints one rbgp-policy-check/1.0 document with located findings. See policy check as a CI and editor gate. |
rustbgpd --migrate-config <pin-legacy|prepare-secure> --offline [--dry-run] <file> | Rewrite the RFC 8212 posture representation in place, offline (Linux only). pin-legacy writes epoch 1 + explicit false. prepare-secure writes epoch 2 + explicit true. --dry-run performs the same validation proof and discards the stage. --offline is an operator assertion, not a daemon probe — stop or quiesce the daemon first. |
rustbgpd --diff <candidate> [<current>] | Compare a candidate file against the current on-disk config (<current> defaults to /etc/rustbgpd/config.toml); print per-section change list with expected reload class and the SIGHUP reload route. Exit 0 = no actionable changes, 1 = actionable changes, 2 = load error. This is a static file-vs-file compare: once the candidate is installed over <current>, it reports no changes. To compare against the running daemon's view, use rbgp config diff. |
rbgp config diff <candidate> | Compare a candidate file against the running daemon's config snapshot, with the same reload-class annotations. Run it on the installed file after a reload to see what did not take effect, such as restart-required edits. Exit 0 = no changes, 2 = changes present, 1 = error. The 1 and 2 codes are the reverse of rustbgpd --diff. |
systemctl reload rustbgpd (or kill -HUP $(pidof rustbgpd)) | Apply the diff. Live fields hot-apply; restart-required fields are pinned and logged at ERROR (the live values are kept). |
The validation pipeline is the same in all three places: TOML parse
→ validate() → ConfigDiff. A config that passes --check will
not error on --diff; a --diff that loads cleanly and does not report
SIGHUP reload route: rejected will not error on reload. A rejected route
means the reload would change nothing; split it as the reason says (see
SIGHUP reload routes).
Observability
Prometheus
The exporter binds at [global.telemetry] prometheus_addr. The same listener
serves /livez and /readyz for orchestrators. Key counters operators watch:
- Session liveness —
bgp_session_established_total,bgp_session_flaps_total,bgp_messages_received_total,bgp_messages_sent_total(all by peer). - Loop / leak detection —
bgp_as_path_loop_detected_total,bgp_rr_loop_detected_total,bgp_otc_routes_blocked_total{peer, reason}(RFC 9234 / ADR-0071),bgp_role_mismatch_total{peer, local_role, remote_role}. Canonicalreasonvalues are documented indocs/reference/operations.md("Ingress rejection / route-leak detection"). - Route processing —
bgp_rib_prefixes{peer, family},bgp_rib_loc_prefixes{family},bgp_max_prefix_exceeded_total. - GR / FIB —
bgp_gr_active_peers,bgp_gr_stale_routes,bgp_fib_routes_installed_total,bgp_fib_kernel_failures_total. - Allocator —
jemalloc_allocated_bytes,jemalloc_active_bytes,jemalloc_resident_bytes,jemalloc_mapped_bytes. Present in builds with thejemallocfeature (the published container image and release tarballs); refreshed at scrape time.allocatedis live application bytes,residentis jemalloc's contribution to RSS — a widening gap between the two is retained-but-unused allocator memory, the first thing to check before suspecting a leak. This jemalloc build takes run-time options from_RJEM_MALLOC_CONF, notMALLOC_CONF; see heap profiling with jemalloc.
Policy filtering visibility — Prometheus. bgp_policy_routes_total {peer, policy, direction, action} attributes each import and export
policy evaluation to the decisive member on Deny. A nonempty chain that
completes without rejection uses the bounded
policy="chain_default_permit" sentinel; policy="inline" remains the
label for an inline Deny or a Permit from an absent / genuinely empty chain.
Initial table dumps, route refreshes, dirty resyncs, and forced outbound
refreshes can increment export counters because they re-evaluate export
policy. Each label identity remains monotonic while its policy/action is
reachable from the installed chain. After a successful policy replacement,
an unreachable identity is retired and goes stale; reinstalling that exact
identity creates a fresh counter starting from zero. Prometheus rate() and
increase() handle the resulting counter reset.
# Routes denied by a named filter on each peer's import side:
rate(bgp_policy_routes_total{direction="import", action="deny"}[5m])An operator policy literally named chain_default_permit remains ordinary
Deny attribution. The sentinel interpretation applies only when
action="permit".
Policy evaluation errors — Prometheus. bgp_policy_eval_errors_total {direction, kind} counts routes denied by the fail-closed evaluation-
error rail (ADR-0103 Decision 4: an action that needs an absent
operand, fuel/loop-cap exhaustion, ...; arithmetic saturates and cannot
fail). These denies also appear in
bgp_policy_routes_total as action="deny"; this counter separates
"the policy said no" from "the policy is broken". The exception is
direction="condition": a conditional advertisement's condition_policy
error denies no route and does not appear in bgp_policy_routes_total; it
can leave the condition unknown. Any nonzero rate deserves a look. For
import and export errors, rbgp policy stats --direction both names the
failing chain, policy, and term (eval_errors count + last_error per
chain), and the rate-limited daemon WARN carries the same blame line. For
condition errors, bgp_conditional_advertisement_condition{state="unknown"}
names the definition, and rbgp rib --prefix P advertised PEER --explain for
a route its advertise_policy selects names the failing condition_policy
and term in the conditional_advertisement step.
# Any policy erroring anywhere is alert-worthy:
sum by (direction, kind) (rate(bgp_policy_eval_errors_total[5m])) > 0Policy filtering visibility — gRPC scalar aggregates. NeighborState
carries four per-peer running totals to give operators a cheap
sanity-check on the labeled Prometheus counter:
| Field | Direction | Scope |
|---|---|---|
import_policy_routes_permitted | import | Daemon lifetime per neighbor — survives session resets and reconfigures; cleared by daemon restart or neighbor deletion. |
import_policy_routes_denied | import | Daemon lifetime per neighbor — same. |
export_policy_routes_permitted | export | Daemon lifetime per neighbor — same. |
export_policy_routes_denied | export | Daemon lifetime per neighbor — same. |
Both directions keep counting across reconnects, like updates_* and
flap_count. For a rate or a per-session window, use increase() over
bgp_policy_routes_total.
The CLI surfaces these in rbgp neighbor <peer> as a Policy Stats
block:
Policy Stats:
Import: permitted=1247 denied=31
Export: permitted=892 denied=0JSON output (rbgp --json neighbor <peer>) carries the same
fields under import_policy_routes_permitted / import_policy_routes_denied
/ export_policy_routes_permitted / export_policy_routes_denied,
elided when zero.
BMP
If [bmp] is configured, rustbgpd opens a TCP session to the BMP
collector and exports per-peer Adj-RIB-In + Peer Up / Peer Down
events. See CONFIGURATION.md for the schema.
gNMI
The gNMI adapter (ADR-0070) exposes Capabilities / Get / Subscribe
telemetry plus the static numbered-neighbor Set subset over the same
sockets as the native API. Native gNMI is registered on
[global.telemetry.grpc_tcp] only when native mTLS is configured; the
[global.telemetry.grpc_uds] listener serves gNMI unconditionally as a
local-only extension. RFC 7951 JSON encoding. See GNMI.md
for the path namespace and supported mutation leaves.
CLI introspection
rbgp is the operator interface for read queries.
rbgp neighbor # list all neighbors
rbgp neighbor 10.0.0.2 # detail
rbgp neighbor 10.0.0.2 --compare 10.0.0.3 # live update-group relationship
rbgp rib # browse Loc-RIB
rbgp bfd # BFD sessions (ADR-0067)
rbgp evpn # EVPN instances + Type 2/3 RIB
rbgp top # live TUI dashboardMost data-oriented read commands support --json for scripting. Commands with
fixed formats, such as metrics, completions, and top, keep their
command-specific output.
For an Established neighbor, rbgp neighbor <addr> reports whether the peer
negotiated Route Refresh, Enhanced Route Refresh, and Extended Messages, plus
the directional maximum BGP message size rustbgpd may send (4096 or 65535
bytes). The JSON fields preserve false as an explicit negotiated result and
omit values an older daemon did not expose; a missing live-session snapshot is
reported separately by negotiation_available.
Package upgrade and rollback
This is the canonical procedure for the release .deb and .rpm packages.
The package scripts create the service user and run systemctl daemon-reload;
on upgrade they deliberately do not stop, start, or restart rustbgpd. An
operator must therefore own the graceful stop and the verification that follows
it. An upgrade over a running daemon leaves the old binary running, and the
package says a restart is required; rbgp health reports the version of the
running daemon, not the file on disk.
Upgrade
-
Read the target release's CHANGELOG section and verify the downloaded package against its release checksum. Config compatibility is not assumed across minor releases.
-
Extract the package into a temporary directory and run the new binary's strict check against the configuration that the service will use:
candidate_pkg=/path/to/rustbgpd_X.Y.Z_amd64.deb # or the .rpm candidate_root=$(mktemp -d) chmod 0755 "$candidate_root" case "$candidate_pkg" in *.deb) dpkg-deb --extract "$candidate_pkg" "$candidate_root" ;; *.rpm) rpm2cpio "$candidate_pkg" | (cd "$candidate_root" && cpio --quiet -id --no-absolute-filenames) ;; *) echo "expected a .deb or .rpm" >&2; exit 2 ;; esac sudo -u rustbgpd \ "$candidate_root/usr/bin/rustbgpd" --check --strict \ /etc/rustbgpd/config.tomlFix every error before proceeding. There is no general schema-migration tool; make release-note field renames by hand, then repeat the candidate check. The one exception is the RFC 8212 posture representation, which
rustbgpd --migrate-config pin-legacy|prepare-secure --offline [--dry-run] CONFIG_PATHrewrites in place (seeCONFIGURATION.md→config_epoch); stop or quiesce the daemon first —--offlineis an operator assertion, not a daemon probe. This check validates candidate config bytes only: it does not inspectruntime_state_diror config-adjacent commit-confirm authority.For the import explain and rejected-route retention capacity limit, older configs with
[policy.explain] cache_sizeor[policy.reject_retention] capacityabove 2,097,152 now fail startup and reload validation, even if that section hasenabled = false. Reduce each named value to at most 2,097,152, then run the new binary's--check --strictagainst the edited file before stopping or restarting the daemon. The defaults remain 4096 and 1024; zero still acts as one.Before any upgrade, use the still-running daemon to run the pre-upgrade diagnostic against the file the new binary will boot:
sudo -u rustbgpd "$candidate_root/usr/bin/rbgp" doctor --pre-upgrade /etc/rustbgpd/config.tomlIt is red, with the next action, while a confirmed transaction is pending, applying, rollback-failed, or ambiguous (confirm or abort it with
rbgp config confirm <id>/rbgp config abort <id>), while a runtime-config settlement owner is still settling, when the file resolves to a different RFC 8212 epoch/posture than the live daemon runs, and whenever the evidence is unavailable or denied. It never confirms, aborts, rewrites, or stops anything. A green result is an observation at one instant (human output saysPre-upgrade observation as of unix <t>; JSON carriesobserved_at_unix_seconds), not a fence: a transaction can still start after it, which is why step 3 stops the daemon and then repeats these checks. See the check reference.When the installed release is v0.64.0 or earlier, also clear retired commit-confirm authority first: a locator-free
commit-confirm-journal.jsonor a retired v2*.commit-confirm-locator.jsonmust be recovered with exactly rustbgpd v0.64.0. Do not proceed while either artifact is present or inaccessible; delete one only after proving the transaction terminal and the current config intended. v0.65.0 and every later release otherwise refuse boot and leave the authority untouched. -
Stop the running daemon cleanly, then confirm it is down:
sudo systemctl stop rustbgpd systemctl is-active rustbgpd # expected: inactiveThe coordinated stop writes the GR restart marker. With GR enabled, the new process can advertise
R=1while sessions rebuild. Families with configured kernel installers (FIB, blackhole discard, EVPN) still advertiseforwarding_preserved = false, while control-plane-only families advertise F=1 under the role rules. This is not a forwarding-continuity guarantee; use a drained route-server pair when traffic continuity matters.If a controller injects routes that must survive the restart at a GR helper (for example blackholes), consider
gr_eor_grace_timeon that neighbor: End-of-RIB then waits for the controller's replay, up to a fixed deadline, instead of leaving with the initial table.With the daemon inactive, repeat the checks that only a stopped daemon can make authoritative: verify the absence of a config-adjacent
*.commit-confirm-locator.json, make any explicit offline posture correction the check called for, then repeat the candidate--check --strictfrom step 2 against the final file after every rewrite. The live diagnostic observed an earlier instant; these post-stop checks are the ones that hold. -
Install the already-checked package:
sudo apt-get install -y "$candidate_pkg" # Debian / Ubuntu sudo dnf install -y "$candidate_pkg" # RHEL / Rocky / AlmaUse the command for the host's package manager, not both. The package marks
/etc/rustbgpd/config.tomlasconfig|noreplace: a locally modified config stays in place. Inspect any replacement candidate reported by the package manager before starting (normally.dpkg-distor.rpmnew):sudo find /etc/rustbgpd -maxdepth 1 -type f \ \( -name '*.dpkg-dist' -o -name '*.rpmnew' \) -print sudo -u rustbgpd /usr/bin/rustbgpd --check --strict \ /etc/rustbgpd/config.toml -
Start and verify the installed version and its operator surfaces:
sudo systemctl start rustbgpd sudo systemctl --no-pager --full status rustbgpd dpkg-query -W -f='${Version}\n' rustbgpd # Debian / Ubuntu rpm -q --qf '%{VERSION}\n' rustbgpd # RHEL / Rocky / Alma sudo -u rustbgpd rbgp health sudo -u rustbgpd rbgp summaryConfirm that the
Version:line fromrbgp health, which the running daemon reports, matches the package you selected.rustbgpd --versionreads the binary on disk, which an unrestarted daemon no longer runs. Wait for every expected session to return toEstablished;systemdactive alone does not prove routing convergence.If the general FIB installer has a v1–v4
fib-owned.json, start a release with the verified v5 rewrite using the existing FIB configuration and runtime state directory before upgrading to one that requires v5. The v5 writer shipped in v0.29.0, but later releases could leave older receipts untouched when routes stayed the same. After startup, check thatfib-owned.jsonin the configuredruntime_state_dirhas"version": 5. Resolve any write or verification error in the daemon log before continuing.
Rollback
A rollback is another compatibility migration, not merely a package command.
Read the target version's release notes. Extract the old package as above and
run its rustbgpd --check --strict against the intended rollback config before
stopping the current daemon. If the old binary rejects the current config,
restore a version-controlled config known to that release, preflight that exact
file with the old binary, and install it only after the stop.
Canonical persistence omits genuinely optional sections while they remain at their defaults. This keeps routine runtime changes from adding unused feature tables to the file. Omitted values use the receiving release's defaults, as they do in hand-written TOML; this is not a cross-release semantic round-trip or downgrade compatibility promise. Once a new feature is configured, its section is retained; renamed fields use the current canonical spelling. An older binary can therefore still reject a newer persisted config. Restoring an older version-controlled config also discards API-driven changes made since that saved copy, so reconcile those changes explicitly before the rollback.
Before downgrading, run rbgp doctor --pre-upgrade against the intended
rollback config (or rbgp config status) and finish a pending confirmed
transaction with rbgp config confirm <id> or rbgp config abort <id>. Do not
delete a live locator to force the downgrade. Once the transaction is terminal,
verify that the config-adjacent *.commit-confirm-locator.json and the retired
locator-free commit-confirm-journal.json are absent. The fixed v3 raw prior
and metadata may remain after a warning-only cleanup failure: once locator
absence is verified they are non-authoritative, and rustbgpd v0.64.0 ignores
those commit-confirm v3-only names. If the target predates either retained
history format, preserve and move the complete config-history/ directory
aside after stopping the daemon and before installing the older binary. In
particular, writable downgrade after a config-history v3 row has been published
is unsupported without moving the directory aside: the older writer ignores
v3, can expose stale rows, and may reuse its sequence. There is no
backfill or down-conversion. Re-upgrade fails closed on duplicate sequences or
an over-cap roster; it cannot make an old writer's collisions trustworthy.
Builds that recognize newer-format history rows (see
ADR-0124) list them as
unreadable and skip recording, with a warning, until they are moved aside.
For a rollback to v0.64.0 that needs an epoch-aware config converted, v0.83.0
was the last release with the Linux downgrade-v0.64 action. Extract both
release binaries. After settling pending transactions and stopping the daemon,
run the v0.83.0 binary on a copy of the config alongside the original so
relative paths resolve from the same directory:
/path/to/v0.83.0/rustbgpd --migrate-config downgrade-v0.64 --offline \
--validator /path/to/v0.64.0/rustbgpd /etc/rustbgpd/rollback.tomlThis converts only the RFC 8212 posture and preserves its effective boolean.
If either release rejects the config, use a saved v0.64.0 config and reconcile
later changes. Run the v0.64.0 binary's --check --strict on the rollback
file before installing it.
For the package rollback, stop the daemon, move history aside if required above, and install the selected older package:
previous_pkg=/path/to/rustbgpd_W.X.Y_amd64.deb # or the .rpm
sudo systemctl stop rustbgpd
# Preserve and move config-history/ aside here if the target cannot read it.
sudo apt-get install -y --allow-downgrades "$previous_pkg" # Debian / Ubuntu
sudo dnf install -y "$previous_pkg" # RHEL / Rocky / AlmaInspect any .dpkg-dist/.rpmnew, install the preflighted compatible config,
then repeat the start and verification commands from the upgrade procedure.
A binary that predates GR marker v3 rejects the v3 marker and cold-starts
without restarting-speaker mode. Do not count on peer route retention across
that boundary; drain the route-server member of a redundant pair first when
continuity matters.
Persistent state on disk
Runtime state lives in runtime_state_dir (default /var/lib/rustbgpd). The
v3 commit-confirm locator instead lives beside the lexical launch config so
startup finds the sole pending authority before trusting candidate contents:
Assign a distinct runtime_state_dir to every concurrently running rustbgpd
daemon. The directory is single-writer runtime state; sharing it between live
processes is unsupported even when their configuration files differ.
| Path | Purpose | Survives restart |
|---|---|---|
gr-restart.toml | Graceful Restart coordination marker. Written on clean shutdown, read on startup to set the R-bit in OPEN. | Yes |
warm-bundle-v1/ | Optional owner-private shutdown checkpoint (manifest.json plus a content-addressed MRT artifact). Published only when warm_cache_checkpoint_on_shutdown = true; not restored on startup. | Yes |
<runtime_state_dir>/commit-confirm-journal.json | Retired locator-free v1/v2 authority; secret-bearing normalized TOML. Its presence makes v0.65.0 and every later release refuse boot untouched. | Until recovery with rustbgpd v0.64.0 or operator-proven terminal deletion |
<runtime_state_dir>/commit-confirm-v3-prior.toml | Fixed v3 raw accepted prior; secret-bearing normalized TOML. Published first. | Until terminal cleanup; safe residue may remain |
<runtime_state_dir>/commit-confirm-v3-metadata.json | Fixed confidential provenance, digest, and file-identity metadata; no raw TOML. Published after the raw prior. | Until terminal cleanup; safe residue may remain |
<absolute lexical config path>.commit-confirm-locator.json | Confidential paths/digests and sole v3 pending boot authority; no raw TOML. Published last and checked before candidate contents. | Until durable unlink and parent sync make the transaction terminal |
config-history/* | Last 20 owner-private v2/v3 JSON rows for rbgp config history; newest is index 0. V2 retains secret-bearing TOML up to 10 MiB; larger accepted snapshots receive at-most-64-KiB metadata-only v3 rows without payloads or source rosters. V3 is permanently rollback-ineligible and independent of commit-confirm recovery. Retired TOML files are ignored and retained. V2 records hash but do not archive external sources and are rollback-eligible only when live sources exactly reproduce recorded provenance. | Yes |
fib-owned.json | FIB ownership receipt — which kernel routes the daemon installed (ADR-0061). Used to drain orphan installs on next start. | Yes |
fib-owned.json.stale | Evidence set aside from a FIB ownership receipt at startup. An unsupported (newer) version or a payload shape that contradicts its version renames the receipt here, and the daemon starts owning nothing. A changed [[fib_tables]] signature copies the receipt here instead and keeps the live fib-owned.json, so unchanged tables keep their owned routes. Diagnostic only; never read back. | Yes; replaced by the next one |
blackhole-owned.json | Exact BLACKHOLE prefix authority. Mode 0600; adoption and deletion require this receipt plus the kernel marker. | Yes |
events.db, events.db-wal, events.db-shm, events.last_id, events.db.stale* | Left behind by the removed durable event history. Current releases never open them; delete them after upgrading. Settle any pending commit-confirm with the previous release first: a retained prior that still holds [event_history] stops the upgraded daemon at startup (see the removal note). | Not used |
crash/panic-*.toml | Panic reports (message, location, thread, binary version, timestamp; never environment or arguments), written through a temporary file and renamed into place. Write-only; the newest 10 are kept, and rbgp doctor includes them in support bundles. An interrupted write can leave a panic-*.toml.tmp file, which retention and rbgp doctor ignore. | Yes |
grpc.sock | gRPC UDS endpoint (if [global.telemetry.grpc_uds] configured). | Recreated on start |
Stop the daemon before deleting blackhole-owned.json; deletion intentionally
preserves all surviving marker rows as foreign. Never share runtime_state_dir
between live instances, and do not downgrade to marker-only adoption without a
drain plan.
The lexical config, metadata, and raw-prior paths resolve to absolute
identities. A writer or present pending object requires daemon-owned real
parents that are not group- or world-writable; locator absence carries no
authority or ordinary-startup storage requirement. Pending and staging files
must be daemon-owned regular files with mode 0600; symlinks and special files
fail closed. Locator unlink plus parent fsync is terminal; only subsequent
verified metadata/raw cleanup and pending-directory fsync are warning-only.
Production reads and writes v3. Retired v1/v2 authority refuses boot untouched;
finish it before upgrade or recover with rustbgpd v0.64.0. Offline --check
does not inspect this runtime state. Before downgrade, finish or abort v3 and
never delete its live locator manually.
Routing state is not restored. The optional shutdown checkpoint contains only eligible post-import-policy Adj-RIB-In views for future use; Loc-RIB, Adj-RIB-Out, and policy evaluation state are not checkpointed, and the current startup path loads none of it. Routing state rebuilds from peer routes after restart. GR and checkpoint state support bounded control-plane restart handling, but do not guarantee forwarding continuity; peers may withdraw or replace routes, and therefore forwarding, until normal convergence.
The config file itself is mutable across runs: neighbor add/delete
operations via gRPC persist back to the config file (see
CONFIGURATION.md → "Config Persistence").
Migrate whole-table family reads
Before the v1 inventory freeze, the scoped RR-only read contract moves to
bounded pages. Replace ListVpnRoutes, ListLabeledRoutes and ListRtcRoutes
with ListVpnRibRoutes, ListLabeledRibRoutes and ListRtcRibRoutes, selecting
view = BEST for the previous winner-only behavior. Replace ListBgpLsRoutes
with ListBgpLsRibRoutes and view = BEST; its old peer_filter becomes
source_peer_filter. Replace ListTopologyNodes and ListTopologyLinks with
ListOrrTopologyNodes and ListOrrTopologyLinks. Replace ListEvpnRoutes
with ListBestEvpnRoutes; select each of Types 1–5 separately for the scoped
EVPN role. The seven whole-table methods remain functional but are deprecated
and outside v1. No RPC is removed. Plain CLI JSON arrays keep their shape;
rbgp builds them from the bounded pages, in the pages' key order.
Follow next_page_token until empty. A table mutation returns ABORTED;
restart the walk with an empty token. Page size 0 defaults to 100; values above
1000 clamp to 1000. Older daemons can return UNIMPLEMENTED for the bounded
methods, so check server support before switching automation. See the
family page contract
for view, filter and peer semantics. Clients using additive EVPN peer filters
must also verify the documented response acknowledgement before using rows.
Sample profiles
The repo ships eleven config profiles, plus a Docker Compose quick-start, under
examples/ covering the standard deployment
shapes. Pick the closest match, copy, edit:
| Profile | File | Use case |
|---|---|---|
| Minimal | examples/minimal/config.toml | Single eBGP peer; dev-friendly, state in /tmp. |
| IX route server | examples/route-server/config.toml | RPKI, Add-Path, dual-stack, per-member policy chains. |
| Route reflector | examples/route-reflector/config.toml | IPv4/IPv6 unicast reflector: client peer group, dynamic client range, non-client peer, Add-Path send, GR/LLGR. |
| MANRS IXP Action 1 | examples/manrs-action1/config.toml | Route server with RPKI-invalid rejection and IRR-derived member filtering; walkthrough in cookbook/manrs-ixp-action1.md. |
| Fabric edge / Linux FIB | examples/linux-edge-fib/config.toml | FIB integration on configured unicast tables; ECMP, weighted multipath. |
| EVPN VTEP leaf | examples/evpn-vtep-leaf/config.toml | Bidirectional VTEP: kernel FDB → Type 2 origination. |
| EVPN RR fabric | examples/rr-evpn-fabric/config.toml | Route Reflector mode, stateless EVI (no [[evpn_instances]]). |
| Route collector | examples/route-collector/config.toml | Passive listener, MRT dumps, BMP export. |
| FlowSpec controller | examples/flowspec-controller/config.toml | gRPC-driven IPv4/IPv6 FlowSpec origination to border routers over FlowSpec-only iBGP sessions. |
| DDoS mitigation | examples/ddos-mitigation/config.toml | FlowSpec injection + RTBH (RFC 7999 BLACKHOLE). |
| Hosting provider | examples/hosting-provider/config.toml | iBGP injector for customer-prefix automation. |
| Docker Compose | examples/docker-compose/ | Quick-start with an FRR peer (gRPC on TCP). |
Each profile validates with rustbgpd --check out of the box; edit
the AS, addresses, and TLS material before deploying.
Security checklist
See SECURITY.md for the full posture document. The
short version for first deployment:
- Bind addresses.
prometheus_addr,grpc_tcp.address,grpc_uds.pathdefault to listening on what the config says. Don't expose the Prometheus or gRPC endpoint to untrusted networks without authentication. - gRPC. Use mTLS in production. Set
[security.grpc] enforcement = "tier"and map principals to roles under[security.grpc.roles](observer/automation/operator) perSECURITY.md; the per-method tier matrix itself is compiled into the daemon (crates/api/src/authz.rs), not configured per-tier in TOML. UDS with a restrictivemodeis fine for single-host operator access. - BGP authentication. TCP-MD5 (RFC 2385) and TCP-AO (RFC 5925) are
both supported. TCP-AO is preferred and needs mainline Linux 6.7 or newer,
or a downstream kernel with TCP-AO backported, built with
CONFIG_TCP_AO=y; see ADR-0062. - RPKI cache transport. RTR (RFC 8210) sessions are TCP with optional
per-cache
md5_passwordortcp_aoauthentication; they are never encrypted. Set a key when the cache supports one; otherwise run caches on loopback or a trusted segment, or tunnel the session (WireGuard, an SSH or stunnel forward). A remote cache reached over an untrusted path is a validation-integrity problem: anything on the path can rewrite the VRPs the daemon validates against. - Firewall. Allow inbound TCP/179 only from configured peer
addresses (or address ranges if using
[[dynamic_neighbors]]). Block everything else.
Troubleshooting
| Symptom | Where to look |
|---|---|
| Session won't establish | rbgp neighbor <addr> + journalctl -u rustbgpd -p warning (or, from JSON logs, journalctl -u rustbgpd -o cat | jq -c 'select(.level=="WARN" or .level=="ERROR")') |
| Routes received but not installed | rbgp rib, then rbgp rib received <peer> --rejected; for the statement-level trace, rbgp policy explain --neighbor <peer> --prefix <CIDR> --direction import (opt-in: needs [policy.explain] enabled = true) |
| Reload didn't change behavior | rbgp config last-reload for pending restart-required fields and the outcome (known_partial lists unapplied identities), then rbgp config diff /etc/rustbgpd/config.toml for edits the running config does not hold, such as restart-required fields; cross-reference the reload matrix. rustbgpd --diff compares two files, so it reports no changes once the new file is installed |
| FIB programming failures | bgp_fib_kernel_failures_total Prometheus counter + journalctl -u rustbgpd -g 'failed to apply general FIB route op' |
| FIB or BLACKHOLE planning stops before kernel access | increase(bgp_dataplane_reconcile_planning_failures_total[10m]) > 0 + matching structured warning; status remains the last successful snapshot |
| EVPN-specific issues | evpn-vtep-troubleshooting.md |
| Anything not above | OPERATIONS.md "Debugging" section |
For deeper investigation, raise the daemon log level globally with the
RUST_LOG environment variable — there is no log-level key in
[global.telemetry]; log_format selects "json" or "text" output and does
not control verbosity. Under systemd, set the
log level in the unit's [Service] section:
[Service]
Environment=RUST_LOG=debugRUST_LOG accepts the usual tracing filter syntax, so a targeted
directive beats a daemon-wide debug — including the per-peer span form
documented in
OPERATIONS.md:
Environment=RUST_LOG=info,[peer{peer_addr=10.0.0.1}]=debugPer-peer verbosity also comes from [[neighbors]] log_level = "debug".
When systemd connects the daemon's stdout to the journal (it sets
JOURNAL_STREAM to stdout's device and inode), each line starts with its
level's syslog priority, which journald strips and stores: ERROR as err
(3), WARN as warning (4), INFO as info (6), and DEBUG/TRACE as
debug (7). This detection is automatic; it needs no configuration and
applies to both log formats. journalctl -p therefore filters by the
daemon's level. Output to a terminal, a file or a pipe has no prefix.
Restart required for the global setting; per-peer log_level is live —
replaced on SIGHUP, so a level edit reaches running sessions without a
restart (see the reload matrix).
Related
OPERATIONS.md— start/stop, reload, upgrade, debugging.CONFIGURATION.md— config reference.reload-matrix.md— per-field reload classification.SECURITY.md— security posture + firewall guidance.INTEROP.md— M-series interop matrix.adr/— architectural decisions per protocol and design choice.
Quickstart
Start and verify a single rustbgpd daemon.
IXP route-server evaluation matrix
One page for an exchange evaluating rustbgpd as a route server: the capabilities IXP evaluations actually score, each with its current status and a link to the config, cookbook, ADR, or measurement receipt that backs it.