rustbgpd

Deploying rustbgpd

Install, operate, and upgrade rustbgpd.

Install, operate, and upgrade rustbgpd.

This is the end-to-end operator runbook: install, run, validate, reload, observe, and upgrade. It's intentionally smaller than the M- series interop matrix in INTEROP.md — that document proves rustbgpd interops cleanly with FRR / BIRD / GoBGP, this one gets your daemon onto your box.

For deeper references:

Status & expectations

rustbgpd is public alpha overall. The exception is the narrow, machine-pinned v1 route-server / route-reflector / controller contract, which covers only its listed config fields, native gRPC signatures, CLI/JSON contracts, and control-plane roles. Unlisted TOML and API surfaces can still change between minor versions. See CHANGELOG.md for migration notes and run rustbgpd --check <new config> against the new binary before swapping it in. FlowSpec controllers should also follow the controller upgrade and rollback procedure to verify server outcomes and view acknowledgements, then reconcile desired rules after restart.

The supported daemon targets are Linux x86_64 and aarch64. Published binaries use a glibc 2.31 baseline. See the canonical platform support contract for CI and non-Linux boundaries.

Suitable for: lab pilots, data-center fabric pilots, IX route-server pilots, automation-heavy control-plane work where the API surface evolution is acceptable. Not yet suitable for fully unattended production deployments where the operator can't react to a CHANGELOG note.

Install

Verified installer

The installer ships as the install.sh release asset from v0.70.0 onward (earlier releases do not include it) and is also available in a source checkout as packaging/install.sh. It resolves latest once, then downloads the selected artifact and its matching per-architecture checksum manifest from that same tag. It refuses unknown architectures, non-glibc hosts, glibc below 2.31, missing checksum rows, duplicate checksum rows, and checksum mismatches before installing.

On Debian/Ubuntu it installs the verified native .deb; on RHEL, Rocky, and AlmaLinux 9+ it installs the verified native .rpm. The package keeps the existing non-replacing configuration behavior. The installer never enables or starts a service; edit the configuration and choose that step yourself.

For a reproducible install from this checkout, inspect and run the installer with an explicit stable tag (vMAJOR.MINOR.PATCH):

less packaging/install.sh
sh packaging/install.sh --tag v0.83.0

Without a source checkout, fetch the published asset:

curl -fsSL https://github.com/lance0/rustbgpd/releases/latest/download/install.sh | sh

Other GNU/Linux distributions must use an explicit prefix or download-only mode. --prefix extracts the verified tarball's existing layout directly into an empty directory; it does not create users, write configuration, or install systemd units. --download-only writes the verified selected artifact and its manifest without installing either.

sh packaging/install.sh --tag v0.83.0 --prefix /opt/rustbgpd-0.83.0
/opt/rustbgpd-0.83.0/rbgp doctor

sh packaging/install.sh --tag v0.83.0 --download-only ./rustbgpd-v0.83.0

Pre-built binary tarball

Tagged releases publish rustbgpd-linux-amd64.tar.gz and rustbgpd-linux-arm64.tar.gz under GitHub Releases. Each ships rustbgpd (the daemon), rbgp (the CLI), rs-config-render (the route-server config renderer), and birdwatcher-adapter (the Alice-LG shim), plus man pages and shell completions under share/:

rustbgpd, rbgp, rs-config-render        binaries
birdwatcher-adapter                     Alice-LG birdwatcher shim
LICENSE-MIT, LICENSE-APACHE, LICENSES.md licenses
rustbgpd.schema.json                    config JSON Schema
share/man/man1/rbgp.1                   CLI man page
share/man/man8/rustbgpd.8               daemon man page
share/completions/rbgp.{bash,zsh,fish}  shell completions
share/systemd/rustbgpd.service          hardened systemd unit
share/systemd/rustbgpd@.service         opt-in per-handle systemd unit
share/systemd/rustbgpd-dataplane.conf   opt-in CAP_NET_ADMIN drop-in
share/monitoring/rustbgpd-overview.json Grafana overview dashboard
share/monitoring/rustbgpd-evpn.json     Alpha EVPN Grafana dashboard
share/monitoring/rustbgpd-alerts.yml    Prometheus alert rules
share/monitoring/rustbgpd-alerts_test.yml promtool rule tests

The binaries are built against glibc 2.31; any distro at or above it works (Debian 11+, Ubuntu 22.04+, RHEL/Rocky/Alma 9+).

The filename is the same on every release; releases/latest/download/ always resolves to the current tag, so this snippet never needs a version bump.

# Pick the right arch.
SUFFIX=linux-amd64    # or linux-arm64
TARBALL=rustbgpd-${SUFFIX}.tar.gz

curl -fL -o "$TARBALL" \
  "https://github.com/lance0/rustbgpd/releases/latest/download/${TARBALL}"
curl -fLO \
  "https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$TARBALL" '$2 == file || $2 == "./" file { print }' \
  "checksums-${SUFFIX}.txt" | sha256sum -c -
tar -xzf "$TARBALL"
sudo install -m 0755 rustbgpd rbgp rs-config-render birdwatcher-adapter /usr/local/bin/
sudo install -m 0644 share/man/man1/rbgp.1 /usr/local/share/man/man1/
sudo install -m 0644 share/man/man8/rustbgpd.8 /usr/local/share/man/man8/
sudo install -m 0644 share/completions/rbgp.bash \
  /usr/share/bash-completion/completions/rbgp

sudo useradd --system --home-dir /var/lib/rustbgpd \
  --shell /usr/sbin/nologin rustbgpd 2>/dev/null || true
sudo install -m 0644 share/systemd/rustbgpd.service \
  share/systemd/rustbgpd@.service /etc/systemd/system/
sudo install -d -m 0755 /etc/rustbgpd
rustbgpd --init-config edge | \
  sudo tee /etc/rustbgpd/config.toml >/dev/null
sudo chmod 0640 /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml

# Kernel-dataplane hosts only; control-plane-only hosts omit this drop-in.
sudo install -d /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 share/systemd/rustbgpd-dataplane.conf \
  /etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.conf

The man pages and completions are also generated on demand by the binaries themselves (rbgp man, rustbgpd --man, rbgp completions bash|zsh|fish), so an installed binary can always regenerate them. Bash and Zsh complete flags at every command level. Fish completes flags only for top-level commands and their direct subcommands; for deeper commands such as rbgp policy chain set-import, it completes the subcommand name but not its flags. All three shells offer --json-lines, --pager, and --json-version only on command paths where those flags apply.

To pin to a specific tag for reproducibility, swap latest for the version, e.g. releases/download/v0.83.0/${TARBALL}. SHA-256 checksums are published alongside each tarball as checksums-${SUFFIX}.txt.

Verify:

rustbgpd --version
rbgp --version
rs-config-render --version

rs-config-render matters only if you run an IXP route server — it turns arouteserver template-context output into rustbgpd configuration (tools/rs-config-render/). It is in the archive either way, so install it now rather than discovering it is missing from a cron refresh later.

For an IXP Manager candidate, run the per-handle rustbgpd@.service instance (its ExecStart reads /var/lib/rustbgpd/<handle>/activation/current/config.toml), then let the renderer publish and settle an immutable generation under that handle:

sudo -u rustbgpd /usr/local/bin/rs-config-render activate \
  --router-handle rs1-ipv4 \
  --candidate /var/lib/rustbgpd/ixp-manager/candidate \
  --runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
  --state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
  --host-state-dir /var/lib/rustbgpd/ixp-manager-host \
  --check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
  --rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
  --activation-command /usr/bin/sudo \
  --activation-arg=-n --activation-arg /usr/bin/systemctl \
  --activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4

Render and activate as the rustbgpd service identity; it must own the private candidate, the per-handle runtime and activation directories, and the shared host-state directory. The runtime directory basename must equal the handle, the activation directory must be exactly <runtime>/activation, and --rbgp-addr must be that runtime's grpc.sock. Authorize that account in sudoers for only the literal /usr/bin/systemctl reload-or-restart rustbgpd@rs1-ipv4. Pre-create the absolute runtime, activation, and host-state directories as non-symlink mode-0700 directories owned by rustbgpd; do not let another process publish or reload during the call. Add --initial only when both no current generation and no reachable daemon exist. Normalized comparison TOML is limited to 4,194,299 bytes (4 MiB minus five encoded-request bytes).

The activation executable must be synchronous. Exit 7 means the candidate was not applied: the executable could not start, or the daemon rejected its reload without runtime effect. The prior link is then restored without another activation and the prior runtime is verified unchanged. The helper accepts a rejection only when rbgp metrics, read before the executable and again after the prior link is restored, shows the same daemon process recorded exactly one SIGHUP outcome, rejected_no_effect, and no runtime-config settlement is in progress. Otherwise, once the executable starts, a nonzero exit, timeout, or unsettled result leaves current on the candidate and returns exit 5; a rejection that cannot be re-proven after the prior link is restored also returns exit 5, with current on the previous generation. Leave retained state untouched and inspect the current private receipt if present; recovery or receipt durability is unproven, and the receipt may be absent or stale when its final write or directory sync failed. rs-config-render status reads that state without changing it and the rs-config-render recover verbs (keep-current, rollback, release-lock, clear; dry run unless --apply) resolve it; the operator procedure for exit 5 is the activation manual-recovery runbook.

To drive the same path directly from IXP Manager v7.4, create a separate mode-0700 candidate directory for the run and store the API key in an absolute mode-0600 regular file owned by rustbgpd:

sudo -u rustbgpd /usr/local/bin/rs-config-render ixp-manager-lifecycle run \
  --ixp-origin https://ixp.example.net --router-handle rs1-ipv4 \
  --api-key-file /var/lib/rustbgpd/ixp-manager/api-key \
  --candidate-dir /var/lib/rustbgpd/ixp-manager/candidate-1 \
  --runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
  --state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
  --host-state-dir /var/lib/rustbgpd/ixp-manager-host \
  --max-prefix-restart-seconds 300 \
  --check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
  --rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
  --activation-command /usr/bin/sudo \
  --activation-arg=-n --activation-arg /usr/bin/systemctl \
  --activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4

The helper uses the exact v7.4 lock, Foil configuration, updated, and release endpoints. HTTPS platform roots are required; redirects, proxies, URL credentials, and path-prefixed origins are disabled. Requests send the API key only in the credential header. A synced private journal precedes each remote effect. Exit 6 means one callback is pending and may be retried with ixp-manager-lifecycle resume; exit 5 means acquisition or activation remains uncertain and no callback is attempted automatically — see the activation manual-recovery runbook. Delivery is at-least-once.

Bound retained activation generations

Every distinct activation leaves an immutable generation under <state-dir>/generations/; activation and lifecycle commands never remove one. Run rs-config-render prune --keep N separately. It reports a dry-run plan unless --apply is present.

Retention reasonProtected generation
currentThe current symlink target
receiptThe last activation receipt's candidate
predecessorThe last activation receipt's previous generation
journalA generation whose render receipt is named by a pending lifecycle journal
keep-NThe N most recently published generations; --keep 0 keeps only referenced generations

Pruning refuses with exit 2 while a host fence or competing host command exists, or when retained receipt or journal state cannot be read safely. The host lock remains held through removal, so recovery state cannot appear between planning and deletion. The tool README's pruning section is authoritative for the complete command, stable output, and cron-safe pattern.

rs-config-render exit codes

Every rs-config-render subcommand shares one exit-code table, mirrored from the tool README:

ExitMeaning
0success — rendered, candidate validated, activated or no-op, or updated delivered
1invalid input — the context or IXP Manager document is unreadable, unparseable, or carries an unknown field, or a site-local file is unreadable (also the generic failure when the final stdout line cannot be written)
2refused — an unsupported knob, an invalid option combination, an unmet precondition (including an unavailable strict checker), no upstream lock acquired, or a definite pre-activation refusal released; nothing is published or activated and no generation, receipt, or journal is left behind (the lifecycle folds a strict-check rejection into this code and leaves that candidate, receipt-less, in its candidate directory for inspection)
3aborted — a generated set is empty or under the plausibility floor (arouteserver mode)
4shape drift — the context's top-level structure drifted from the pinned fingerprint; pass --allow-shape-drift to proceed (arouteserver mode)
5manual recovery — a human is needed: the activation effect is uncertain (current stays on the candidate, or on the previous generation when a no-effect rejection could not be re-proven after restoring it) or a recover --apply step did not complete (current is wherever that step left it — on the rollback target after a rollback that did not settle); retained state and any upstream lock are kept and no callback is issued; inspect with status before acting
6callback pending — one durable updated or release callback is undelivered; run ixp-manager-lifecycle resume
7rolled back — the candidate was not applied: the activation command never started, or the daemon rejected its reload without runtime effect; the prior generation is restored and proven and the lock is released; retrying is safe
8output unusable — the candidate directory is not an absent or empty private directory (IXP Manager mode), could not be created or written (arouteserver mode), or a prune --apply removal failed
9strict check failed — rustbgpd --check --strict ran and rejected the rendered IXP Manager candidate (the only path to this code); its files stay in the candidate directory without a receipt

Debian / RPM packages

Tagged releases also publish native packages built with nfpm from the same binaries as the tarballs (packaging/nfpm.yaml):

  • rustbgpd_X.Y.Z_{amd64,arm64}.deb — Debian 11+, Ubuntu 22.04+
  • rustbgpd-X.Y.Z-1.{x86_64,aarch64}.rpm — RHEL/Rocky/Alma 9+
# Debian / Ubuntu
curl -fLO https://github.com/lance0/rustbgpd/releases/latest/download/rustbgpd_X.Y.Z_amd64.deb
sudo apt-get install -y ./rustbgpd_X.Y.Z_amd64.deb

# RHEL / Rocky / Alma
sudo dnf install -y \
  https://github.com/lance0/rustbgpd/releases/download/vX.Y.Z/rustbgpd-X.Y.Z-1.x86_64.rpm

The package installs the four binaries (rustbgpd, rbgp, rs-config-render, and birdwatcher-adapter) to /usr/bin, the hardened systemd singleton and opt-in per-handle template units (see systemd below — ExecStart pointed at /usr/bin), man pages and shell completions, and creates the rustbgpd system user (imperatively at install time, plus a declarative sysusers.d file). A starter config lands at /etc/rustbgpd/config.toml (mode 0640 root:rustbgpd, never overwritten on upgrade) — the minimal profile with its state paths set to /var/lib/rustbgpd. Version-matched monitoring payloads land under /usr/share/doc/rustbgpd/monitoring/. Then:

sudo $EDITOR /etc/rustbgpd/config.toml   # set ASN, router_id, neighbors
sudo systemctl enable --now rustbgpd

The kernel-dataplane capability drop-in ships as documentation at /usr/share/doc/rustbgpd/examples/rustbgpd-dataplane.conf; copy it into /etc/systemd/system/rustbgpd.service.d/ only for deployments that program the kernel (see below). Config-mutating RPCs (rbgp neighbor add, gNMI Set) need /etc/rustbgpd writable by the service user; the packaged default (0750 root:rustbgpd, read-only for the daemon) supports the SIGHUP-reload workflow — chown rustbgpd /etc/rustbgpd /etc/rustbgpd/config.toml if you want runtime mutation.

Removing the package (apt-get remove/purge, dnf remove) stops and disables rustbgpd and every rustbgpd@ instance before the binaries are deleted, so the unit's ExecStopPost still releases any EVPN core-isolation links. The stop is graceful and withdraws the daemon's routes from its peers. /var/lib/rustbgpd and the rustbgpd user are kept; purge also deletes /etc/rustbgpd/config.toml on Debian and Ubuntu.

Verifying release artifacts

Each architecture has a checksums-<arch>.txt manifest covering its tarball and native packages. Select the exact downloaded filename before passing the row to sha256sum; checking the whole manifest fails when the other package formats have not also been downloaded:

SUFFIX=linux-amd64
ARTIFACT=rustbgpd-linux-amd64.tar.gz
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/${ARTIFACT}"
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$ARTIFACT" '$2 == file || $2 == "./" file { print }' \
  "checksums-${SUFFIX}.txt" | sha256sum -c -

This detects a download that differs from the manifest published with the GitHub release; it does not independently authenticate GitHub. The committed Cargo.lock is the exact Rust dependency inventory for a tagged source tree. Release binaries and containers are provided under the repository's license terms and warranty disclaimers.

Monitoring payloads

Every release archive carries its version-matched Grafana dashboards, Prometheus alert rules, and promtool test suite under share/monitoring/. Native packages put the same files under /usr/share/doc/rustbgpd/monitoring/. Keep rustbgpd-alerts.yml beside rustbgpd-alerts_test.yml: the suite deliberately uses the relative rule_files entry rustbgpd-alerts.yml.

Validate the shipped rules before loading them:

# From an extracted release tarball:
(cd share/monitoring && promtool test rules rustbgpd-alerts_test.yml)

# Or from a native package:
(cd /usr/share/doc/rustbgpd/monitoring && \
  promtool test rules rustbgpd-alerts_test.yml)

Copy or reference rustbgpd-alerts.yml from Prometheus's rule_files configuration, then import rustbgpd-overview.json and, when operating EVPN, the Alpha rustbgpd-evpn.json dashboard in Grafana. The Grafana guide supplies the matching scrape job and dashboard setup; keep the payloads from one rustbgpd release together so rules and metrics do not drift across versions.

From source

# Prerequisites: Rust ≥ 1.95, protobuf-compiler
sudo apt-get install -y protobuf-compiler   # Debian/Ubuntu
git clone https://github.com/lance0/rustbgpd
cd rustbgpd
# Builds exactly what a deployment ships: daemon, rbgp, the route-server
# config renderer, and Birdwatcher adapter. (`--workspace` would also build
# dev/bench helpers you don't need.)
cargo build --release -p rustbgpd -p rustbgpctl -p rs-config-render -p birdwatcher-adapter
sudo install -m 0755 \
  target/release/{rustbgpd,rbgp,rs-config-render,birdwatcher-adapter} \
  /usr/local/bin/

Container image

The current release workflow can build and runtime-verify images natively on Linux amd64 and arm64, then publish a single multi-arch (amd64+arm64) manifest on a tagged release. This does not backfill existing images: :0.68.0 and :0.68 remain amd64-only. :latest follows the newest non-prerelease release and inherits that release's platform set. The workflow configuration produces three tag flavors with docker/metadata-action in .github/workflows/container.yml:

TagResolves toUpdates on
:X.Y.Zexact version and its published platform setnothing (immutable)
:X.Ylatest patch in the X.Y minoreach X.Y.Z release
:latestlatest non-prerelease release, including its platform seteach minor or patch release

Major-minor is the usual operator default — auto-receives bug-fix releases but pins against minor-version churn. The examples in this document use :latest so the tag stays copy-pasteable across releases. Substitute the major-minor tag of the series you standardize on:

docker pull ghcr.io/lance0/rustbgpd:latest

The published image is the Dockerfile's default runtime target: a lean daemon with rbgp and birdwatcher-adapter, running as a nonroot rustbgpd user. No dev/test/bench helpers are included. rs-config-render is deliberately left out too: the IXP refresh loop is a host-side cron job that runs arouteserver and the renderer next to each other and hands the daemon a finished config directory, so the renderer belongs on the host, not in the daemon's image. Install it from the tarball. If you'd rather build the image locally:

docker build -t rustbgpd:latest .

The development image — adds evpn-tester / evpn-monitor, iproute2, and the interop start script, and runs as root for lab plumbing — is the dev target. It's what the M-series interop suite and the soak harnesses run against:

docker build --target dev -t rustbgpd:dev .

Birdwatcher adapter

Native packages install birdwatcher-adapter at /usr/bin/birdwatcher-adapter; the production container installs it at /usr/local/bin/birdwatcher-adapter. For a same-host deployment, point it at the daemon's absolute Unix socket and run it under a supervisor as the rustbgpd user:

sudo -u rustbgpd birdwatcher-adapter \
  --grpc-addr unix:///var/lib/rustbgpd/grpc.sock \
  --listen 127.0.0.1:8080

TCP remains supported with --grpc-addr http://127.0.0.1:50051. The adapter may start before the daemon/socket: REST requests return 502 Bad Gateway while gRPC is unavailable and recover without restart. No service unit is installed; use the supervisor and lifecycle policy appropriate for the Alice-LG deployment. See the adapter README for bearer-token and endpoint details. This convenient owner-only socket authenticates as the operator-tier local-operator; use a dedicated token-authenticated listener mapped to observer when least privilege is required.

systemd

A hardened unit lives at examples/systemd/rustbgpd.service. It runs the daemon unprivileged by default: a dedicated rustbgpd system user, the standard sandbox set (NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, StateDirectory / RuntimeDirectory), and a capability set of exactly CAP_NET_BIND_SERVICE (bind port 179 as non-root). Deployments with interface-only neighbors also need the opt-in CAP_NET_RAW drop-in.

Releases also ship the opt-in rustbgpd@.service for IXP Manager. Literal %i is the exact router handle: it selects /var/lib/rustbgpd/%i/activation/current/config.toml, a private per-handle runtime/UDS, and no writable access to the shared lifecycle fence. Pre-create each exact handle and the one shared fence directory before the first lifecycle run; authorize and invoke only literal instances, never a wildcard sudoers rule:

handle=rs1-ipv4
sudo install -d -m 0700 -o rustbgpd -g rustbgpd \
  "/var/lib/rustbgpd/$handle" "/var/lib/rustbgpd/$handle/activation" \
  /var/lib/rustbgpd/ixp-manager-host
sudo systemctl enable "rustbgpd@$handle"

Use distinct runtime/activation/UDS paths for every handle and the same /var/lib/rustbgpd/ixp-manager-host for all lifecycle commands on that host.

Notes on the sandbox:

  • CAP_NET_BIND_SERVICE lets the daemon bind port 179 without running as root. Nothing else is granted by default — in particular not CAP_NET_ADMIN, so the unprivileged unit cannot program the kernel.

  • ProtectSystem=strict + explicit ReadWritePaths confines the daemon to /var/lib/rustbgpd (state) and /etc/rustbgpd (config).

  • ExecStart passes --runtime-state-dir /var/lib/rustbgpd; the daemon refuses to start when that differs from the config's runtime_state_dir, so change both together. ExecStopPost= runs rustbgpd --release-core-isolation --runtime-state-dir /var/lib/rustbgpd after every stop, crash or failed start: it raises CE-facing links that an EVPN core-isolation hold left down, needs CAP_NET_ADMIN only when there is something to release, and exits 0 when there is nothing to release.

  • ExecReload=kill -HUP is the supported reload path. systemctl reload returns once the signal is sent; automation that needs the outcome runs rbgp reload, which sends the same signal, waits for the reload to settle and exits non-zero unless it completed. See the reload matrix for which fields hot-apply vs. need a restart.

  • Type=notify is load-bearing. The daemon sends READY=1 only after every configured gRPC listener is bound, the configured peer roster is installed, and BGP ingress is active — the same boundary /readyz cannot go green before — so systemctl start and units ordered After=rustbgpd.service wait for a daemon that can accept sessions, not for a forked process. A gRPC bind failure prevents peer registration, BGP ingress, telemetry, dial-out, and readiness publication before entering shortened coordinated teardown. STOPPING=1 marks the start of coordinated shutdown. TimeoutStartSec=10min bounds initial roster installation before READY=1.

  • WatchdogSec=5min restarts a daemon whose core actors have stopped answering. Every 10 seconds the daemon asks the PeerManager and the RIB actor for an answer, waits up to 5 seconds, and sends WATCHDOG=1 when both answered. This is a liveness sample, not /readyz:

    • A RIB that answers "stalled" from inside a long policy transition, selection-deferral release or peer teardown is working, not dead. /readyz returns 503 for it; the watchdog keeps pinging.
    • Only silence withholds the ping: an actor that does not answer within 5 seconds, a closed channel or a dropped reply. A ping that the notify socket refuses counts the same way. The daemon logs a WARN, increments bgp_systemd_watchdog_pings_skipped_total and samples again one second later, so a silent actor is asked every 6 seconds (5 seconds of waiting, then the 1-second retry).
    • The sampler runs on its own thread. It does not depend on the telemetry listener being configured, and a busy daemon worker cannot delay it.
    • The sample ignores the shutdown gate, so pings continue through coordinated shutdown for as long as the actors answer. Once the peer manager starts draining sessions it stops answering, so the counter moves until the process exits. That cannot abort a stop: systemd stops enforcing the watchdog when it receives STOPPING=1, and TimeoutStopSec bounds the stop from then on.

    The watchdog does not bound how long the RIB may keep answering "stalled". A unit of work that keeps making progress is never aborted, however long it runs. Alert on it instead: the daemon samples the RIB every 5 seconds and exports bgp_rib_stalled (1 while the latest sample was a stalled answer). The shipped BgpReadinessRibStalled rule fires when it reads 1 on every evaluation for ten minutes. It needs only a /metrics scrape, no /readyz probe. Restart by hand only if the work is not finishing.

    PID 1 independently sends SIGABRT five minutes after the last ping, and Restart=on-failure covers that expiry. Measured from the moment an actor stops answering, that is about five minutes: 290 to 300 seconds when answers were prompt, and 285 to 305 seconds at the edges. A sample that took almost 5 seconds to answer makes the last ping up to 15 seconds old, and an actor that stops while the other's answer to the same sample is still on its way lets one more ping out up to 5 seconds later. A stopped process (SIGSTOP, a debugger) is aborted 285 to 300 seconds after it stops. With another WatchdogSec the sample period is a quarter of it and the sample deadline an eighth, capped at 10 and 5 seconds, so the abort comes between five eighths and nine eighths of the timeout after an actor stops.

  • TimeoutAbortSec=2min bounds the wait after that SIGABRT. The daemon does not handle the signal, so the kernel writes a core dump and the restart waits for it; every session is already down. Two minutes is enough for a multi-gigabyte core, and systemd truncates a larger one with SIGKILL. Without the setting systemd would wait TimeoutStopSec, 32 minutes. Raise it if complete cores of a very large table matter more than restart time. It needs systemd 243 or newer; older versions ignore it with a warning.

    Without NOTIFY_SOCKET — a foreground run or Docker — every notification is a silent no-op. A native unit opting out must disable all inherited notify behavior:

    [Service]
    Type=simple
    NotifyAccess=none
    WatchdogSec=0

    The container unit keeps Type=simple because Docker, not the daemon, is its main process.

  • Restart=on-failure is load-bearing. The daemon exits 0 only on an operator-initiated shutdown (SIGINT/SIGTERM, the Shutdown RPC) and 1 on a component failure it cannot recover from in place. Startup exits immediately if legacy BGP mode binds neither family, explicit listen_addresses cannot bind every configured endpoint, or a configured prometheus_addr health listener cannot bind, or any configured gRPC listener cannot bind. An unexpected RIB manager, peer manager, RPKI subsystem task, BGP listener task, inbound accept-forwarding task, or injection lease expiry task exit instead runs the coordinated peer teardown before exit 1. These components are not respawned in place; the supervisor is the recovery path. The gRPC server and the metrics/readiness server are the exception: the daemon restarts them in place and keeps BGP running (see gRPC server dies unexpectedly and Metrics/readiness server exits unexpectedly). Exit 70 is also a failure: it is the runtime-config settlement watchdog's fail-stop recovery request and must remain restartable. RestartSec=5 retries a transient failure, while StartLimitBurst=5 and StartLimitIntervalSec=10min stop a deterministic failure from flapping forever. After correcting config authority or permissions, recover a rate-limited unit with systemctl reset-failed rustbgpd followed by systemctl start rustbgpd. An explicit systemctl stop suppresses restart, and TimeoutStopSec=32min gives an already-owned mutation longer than its 30-minute watchdog plus five-second terminal grace to settle or fail-stop. Nothing else in the stop path needs that long: waits outside settlement ownership are bounded in seconds, and a second SIGTERM or SIGINT skips the ones that have no deadline, so TimeoutStopSec does not need to change. A second signal while a mutation is still settling instead forces the watchdog's fail-stop at once (fence_reason="operator_forced", exit 70 after the grace). The supervisor consequence depends on how the stop was requested: raw signals to a running unit end in exit 70, an unclean exit code that Restart=on-failure restarts, while a unit stopped explicitly with systemctl stop is never restarted automatically whatever its exit status. Recovery from the persisted transaction runs on the next actual start either way. rbgp doctor reports listener failures through its bgp.listener check.

Installation

The Debian/RPM packages perform all of the below on install (unit at /lib/systemd/system, ExecStart at /usr/bin). The tarball commands above use its share/systemd/ tree. The following paths are only for a source checkout:

sudo useradd --system --home-dir /var/lib/rustbgpd \
  --shell /usr/sbin/nologin rustbgpd
sudo install -m 0644 examples/systemd/rustbgpd.service \
  /etc/systemd/system/rustbgpd.service
sudo install -d -m 0755 /etc/rustbgpd
sudo install -m 0640 examples/minimal/config.toml /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml   # readable by the service user
$EDITOR /etc/rustbgpd/config.toml    # set ASN, router_id, neighbors
sudo systemctl daemon-reload
sudo systemctl enable --now rustbgpd

Interface-only neighbors: opt-in privilege drop-in

When at least one neighbor has interface without address, grant CAP_NET_RAW for ICMPv6 discovery, including with [global.unnumbered] router_advertisements = false. Static link-local neighbors with an explicit address and RS/RR deployments without interface-only neighbors do not need this capability. The base unit keeps its existing capability set; install the separate example from the source checkout:

sudo mkdir -p /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 examples/systemd/rustbgpd-unnumbered.conf \
  /etc/systemd/system/rustbgpd.service.d/rustbgpd-unnumbered.conf
sudo systemctl daemon-reload
sudo systemctl restart rustbgpd

The drop-in adds CAP_NET_RAW to AmbientCapabilities and CapabilityBoundingSet. Install the dataplane drop-in alongside it when the configuration also programs the kernel. For a per-handle unit, place the capability drop-in under the exact rustbgpd@<handle>.service.d/ directory.

A peer RA with a non-zero Router Lifetime can install a kernel default route independently of BGP if the host accepts RA defaults. Disable those defaults on dedicated peering ports (for example, net.ipv6.conf.<port>.accept_ra=0) when routing must follow BGP. rustbgpd receives discovery packets on its raw socket regardless of that sysctl. Its own RAs also carry a non-zero Router Lifetime (three times ra_interval_secs, 30 seconds by default), so neighboring hosts must apply the same RA-default policy when their routing should follow BGP. See the interface-only configuration.

Kernel dataplane: opt-in privilege drop-in

If — and only if — the config programs the kernel ([[fib_tables]] per ADR-0061, install_blackhole_discard, or the EVPN VTEP/IRB dataplane), the daemon needs CAP_NET_ADMIN for rtnetlink route / nexthop programming, VXLAN + bridge FDB writes, and the RTNLGRP_NEIGH subscription behind local-MAC learning. Grant it via the shipped drop-in examples/systemd/rustbgpd-dataplane.conf instead of editing the base unit — systemd merges capability sets, so the drop-in adds CAP_NET_ADMIN on top of the unprivileged default:

sudo mkdir -p /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 examples/systemd/rustbgpd-dataplane.conf \
  /etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.conf
sudo systemctl daemon-reload
sudo systemctl restart rustbgpd

RR-only boxes should not install this drop-in — the unprivileged default is the point.

Dataplane host caveat: if rustbgpd programs the kernel on a host that also runs systemd-networkd, set ManageForeignRoutes=no, ManageForeignRoutingPolicyRules=no, and (where supported) ManageForeignNextHops=no in networkd.conf. The defaults are yes, and networkd will delete routes and next-hop groups it did not create — including rustbgpd's, especially across a daemon restart (ADR-0079). Co-residency with another daemon that claims proto bgp kernel state (e.g. FRR zebra) is unsupported for the same reason.

Reserved nexthop-ID ranges (EVPN dataplane)

When the EVPN VTEP/IRB dataplane is enabled, rustbgpd exclusively owns four kernel nexthop-ID ranges, distinguished by the ID's high nibble (deliberately offset from FRR's 0x1000/0x2000 tags so both daemons can't collide on NLM_F_REPLACE):

RangeUse
0x3000_0000–0x3FFF_FFFFL2 per-VTEP FDB nexthop members (ADR-0059 aliasing ECMP)
0x4000_0000–0x4FFF_FFFFL2 FDB nexthop groups (ADR-0059)
0x5000_0000–0x5FFF_FFFFL3VXLAN per-VTEP nexthop members (all-active Type 5)
0x6000_0000–0x6FFF_FFFFL3VXLAN FDB nexthop groups (all-active Type 5)

This is a deployment contract: no co-resident netlink writer may create nexthop objects with IDs in these ranges (ip nexthop add id …, another routing daemon, orchestration tooling). rustbgpd adopts objects it finds there across restarts and garbage-collects the ones nothing references — an unrelated agent's object parked in the range would be treated as rustbgpd state.

rustbgpd validates the shape of every in-range object it dumps at startup adoption and during drift recovery (FDB flag, member vs group structure matching the tag). An in-range object that does not look like anything rustbgpd writes fails closed per ID: the object is left untouched, excluded from adoption and from every reap/delete path, its ID is quarantined so the allocator can never hand it out (and thus never NLM_F_REPLACE over it), and the violation is logged once and counted on the evpn_foreign_nhid_range_conflicts_total Prometheus counter. A nonzero counter means a co-resident writer is violating this contract — find and remove that writer; the quarantined IDs are not reclaimed until a daemon restart after the foreign objects are gone.

Note this shape check cannot distinguish a foreign object that is byte-for-byte identical to what rustbgpd writes; stamping nh_protocol on owned nexthop objects as a provenance marker is a possible future defense-in-depth, not a substitute for this contract (any netlink writer can set the same protocol value — it is provenance, not kernel-enforced exclusivity).

Operational checks

systemctl status rustbgpd            # is it running?
journalctl -u rustbgpd -f            # follow logs
systemctl reload rustbgpd            # SIGHUP → config reload
sudo rbgp reload                     # SIGHUP, wait, print the outcome receipt
rbgp config last-reload              # outcome of the latest settled reload
rbgp config diff /etc/rustbgpd/config.toml  # installed file vs running config
systemctl restart rustbgpd           # full restart (state drained)

Docker / Docker Compose

A worked starting point at examples/docker-compose/ brings up rustbgpd alongside an FRR peer that advertises sample prefixes — no real routers required.

cd examples/docker-compose
docker compose up -d --build
docker compose exec rustbgpd \
  rbgp -s http://127.0.0.1:50051 neighbor
docker compose down

For your own deployment:

  • Image name and command: published GHCR version tags have no leading v: use ghcr.io/lance0/rustbgpd:0.83.0, not ghcr.io/lance0/rustbgpd:v0.83.0. The older :0.68.0 image is amd64-only; it is not backfilled when a later release publishes multiple platforms. The production image runs as uid/gid 999 — pinned in the Dockerfile runtime stage (useradd --uid 999 --gid 999 and USER 999:999), not allocated by the base image, and asserted against the built image by the container workflow — and its default command is exactly rustbgpd /etc/rustbgpd/config.toml. If the config is mounted under another container filename, pass that filename explicitly after the image, for example rustbgpd /etc/rustbgpd/router.toml.

  • Bridge networking and state: mount a volume at the daemon's runtime_state_dir so the GR restart marker and FIB and BLACKHOLE ownership receipts survive container restarts. The daemon runs as uid/gid 999 and rewrites its config in place, so both host directories must be owned by that uid and the config must already exist — the image's default command is rustbgpd /etc/rustbgpd/config.toml and it will not create one:

    # One-time host prep.
    sudo install -d -o 999 -g 999 /etc/rustbgpd /var/lib/rustbgpd
    docker run --rm ghcr.io/lance0/rustbgpd:latest \
      rustbgpd --init-config edge > config.toml
    # Edit config.toml for your ASN, router ID, peers, and policy.
    sudo install -o 999 -g 999 -m 0600 config.toml /etc/rustbgpd/config.toml
    
    docker run --rm -d \
      --name rustbgpd \
      --network=bridge \
      --stop-timeout=1920 \
      -v /etc/rustbgpd:/etc/rustbgpd \
      -v /var/lib/rustbgpd:/var/lib/rustbgpd \
      -p 179:179 \
      -p 9179:9179 \
      --ulimit nofile=65536:524288 \
      ghcr.io/lance0/rustbgpd:latest

    This recipe succeeds because Docker sets net.ipv4.ip_unprivileged_port_start=0 inside its default bridge network namespace. Port 179 is therefore not privileged there; no capability flag is responsible for the bind. Add NET_ADMIN only when the container actually programs its network namespace. The bridge path is the supported choice when the daemon does not need to bind a particular host/peering-LAN address.

    The edge profile is used because its runtime_state_dir, grpc_uds path, and 0.0.0.0 metrics bind are already the container forms; the lab profile's /tmp state directory and 127.0.0.1 metrics bind both need editing first.

Host networking and privileged BGP ports

Host networking is the second supported container path, and is required when listen_addresses must name a host/peering-LAN address, a multihop session pins its source, or the daemon must program the host network namespace. It does not inherit Docker's bridge-netns low-port setting:

Docker network modeEffective net.ipv4.ip_unprivileged_port_startImage uid 999 can bind :179
default bridge0 in the container namespaceyes
--network=hostthe host value (commonly 1024)no when the host value is above 179

--cap-add=NET_BIND_SERVICE is not sufficient for the image's uid 999 in host mode. Docker does not provide that capability as an ambient capability to the non-root image process, so it is unavailable after the uid 999 exec. This differs from the bare-binary systemd unit above, whose AmbientCapabilities=CAP_NET_BIND_SERVICE is effective and remains correct.

Use exactly one of these deployment paths:

  1. Keep the default bridge network and published ports when address pinning is unnecessary; the image continues to run as uid 999.

  2. Use --network=host with --user=root and bind a root-owned host state directory at /var/lib/rustbgpd:

    sudo install -d -o root -g root -m 0755 /etc/rustbgpd
    sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd
    sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml
    
    docker run --rm -d \
      --name rustbgpd \
      --network=host \
      --user=root \
      --cap-drop=ALL \
      --cap-add=NET_BIND_SERVICE \
      --stop-timeout=1920 \
      --ulimit nofile=65536:524288 \
      --mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \
      --mount=type=bind,source=/var/lib/rustbgpd,target=/var/lib/rustbgpd \
      ghcr.io/lance0/rustbgpd:0.83.0

Do not omit the host state bind when overriding the image user. The directory baked into the image is owned by uid 999; a root daemon expects uid 0. The runtime-state owner guard treats that mismatch as unsafe and refuses to start with refusing to start: runtime_state_dir must be a directory owned by the daemon user and not group- or world-writable. v0.69.0 and later images also reject the default gRPC UDS listener at startup. A root-owned, non-group/world-writable bind mount makes the authority match the running uid. The socket parent must allow the daemon read/write/search access, with no symlinks or untrusted writable ancestors; see Unix socket path integrity.

Adding NET_BIND_SERVICE to uid 999 is not a third supported host-network recipe. If changing the host's low-port sysctl is part of a separately managed host policy, validate that policy outside this unit; the shipped container guidance does not mutate the host sysctl.

The root host-network path drops Docker's entire default capability set, then adds back only NET_BIND_SERVICE for port 179. Root makes that explicitly granted capability usable; applying the same add-cap flag to uid 999 remains ineffective because Docker supplies no ambient capability to that non-root exec.

The command above and the systemd unit below are control-plane defaults. A configuration that programs the host FIB or interfaces also needs --cap-add=NET_ADMIN on the same root host-network path, as described in kernel dataplane privileges; that opt-in does not replace the root-user and root-owned-state requirements.

Interface-only neighbor discovery also needs --cap-add=NET_RAW on that root host-network path, even when router_advertisements = false. The flag does not supply an ambient capability to the image's default non-root process. Static link-local neighbors with an explicit address do not need NET_RAW.

systemd-supervised host-network container

examples/systemd/rustbgpd-container.service generalizes the host-network recipe above. It keeps docker run attached so the daemon's exit status drives Restart=on-failure, maps systemctl reload to SIGHUP, gives Docker the full 32-minute settlement grace and systemd a 33-minute outer margin, and makes the registry pull best-effort so an explicitly selected cached image remains usable during a registry outage. A missing or empty RUSTBGPD_IMAGE fails closed before any Docker command runs.

Install the unit and its required image environment file after creating the root-owned state directory:

sudo install -d -o root -g root -m 0755 /etc/rustbgpd
sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd
sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml
sudo install -m 0644 examples/systemd/rustbgpd-container.service \
  /etc/systemd/system/rustbgpd-container.service
sudo tee /etc/rustbgpd/rustbgpd-container.env >/dev/null <<'EOF'
RUSTBGPD_IMAGE=ghcr.io/lance0/rustbgpd:0.83.0
EOF
sudo chmod 0644 /etc/rustbgpd/rustbgpd-container.env
sudo systemctl daemon-reload

The overridable host defaults are RUSTBGPD_CONFIG_DIR=/etc/rustbgpd and RUSTBGPD_STATE_DIR=/var/lib/rustbgpd; both are bind-mounted at their standard container paths. RUSTBGPD_CONFIG_FILE=/etc/rustbgpd/config.toml selects the file inside the read-only config mount. Put any overrides beside RUSTBGPD_IMAGE in the environment file. An upgrade is one image-tag edit followed by systemctl restart rustbgpd-container; version tags still omit the leading v. RUSTBGPD_IMAGE is required and must be non-empty; the unit checks it before its best-effort pull.

The unit deliberately mounts the config directory read-only. Runtime mutation (rbgp neighbor add, policy edits, gNMI Set, and rbgp config apply) is therefore excluded: those calls cannot persist and are rejected. Supporting them requires an explicit change to the mount and root-owned config-directory write contract; the shipped unit is for externally managed config plus SIGHUP.

Before the first start, and before restarting after an image or config change, validate the exact externally managed config directory and selected image without touching the running container:

docker run --rm \
  --user=root \
  --cap-drop=ALL \
  --mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \
  ghcr.io/lance0/rustbgpd:0.83.0 \
  rustbgpd --check --strict /etc/rustbgpd/config.toml

Only enable the service after that check succeeds:

sudo systemctl enable --now rustbgpd-container

The image's declared healthcheck remains active under systemd supervision and runs rbgp health --liveness (gRPC handler liveness, not core-actor readiness) against the default state-directory UDS. An unhealthy result remains observable with docker inspect --format '{{.State.Health.Status}}' rustbgpd, but Docker health status does not terminate the attached process and therefore does not drive systemd's Restart=on-failure. If the config moves the socket, set RUSTBGPD_ADDR for the container by extending the unit's docker run command.

  • File descriptors: --ulimit nofile sets the ceiling. rustbgpd raises its soft limit to the hard limit at startup, so the hard limit must clear the 4096 floor rbgp doctor enforces; peers exhaust descriptors at scale.

  • Writable config directory: mount /etc/rustbgpd read-write and make it owned by the daemon user (chown the host directory to the container's rustbgpd uid) if you use runtime mutation — rbgp neighbor add, policy edits, gNMI Set, rbgp config apply. Config persistence rewrites the file with a temp-file + rename, so the directory, not just the file, has to be writable, and every mutating RPC is rejected without it. Mount it :ro only when the config is managed entirely from outside and reloaded with SIGHUP.

  • Settlement fail-stop: --stop-timeout=1920 gives the daemon a 32-minute Docker stop grace; the shipped container unit's TimeoutStopSec=33min is the supervisor margin around that operation. A failure to create the initial persistence stage rejects cleanly before runtime mutation. After runtime mutation begins, a failed rename is NotPublished: only complete acknowledged compensation returns cleanly; otherwise known divergence fences recovery. A post-rename directory-fsync failure is PublicationAmbiguous: the complete visible candidate is adopted without rollback, and restart selects durable authority. A read-only bind-mounted config file can have a writable parent yet still fail replacement as NotPublished. A fenced daemon marks readiness unavailable, closes persisted-mutation admission, and exits 70 within five seconds; a silent owner is fenced at 30 minutes and exits five seconds later. This standalone command has no restart policy; use bounded supervisor retries and inspect durable config authority before recovery.

  • Health: the image runs rbgp health --liveness every 30 seconds. Success means only that the authenticated gRPC handler answered; it does not check core-actor readiness, BGP convergence, or forwarding. This is an intentional change from the previous image probe, which used GetHealth and checked core actors. To retain that behavior, start the container with --health-cmd='rbgp health'. Ordinary rbgp health keeps its detailed output and readiness behavior. The liveness RPC requires only the read listener tier; credentials and principal-role checks still apply. Failed, refused, or unreachable probes exit non-zero. The default endpoint is unix:///var/lib/rustbgpd/grpc.sock; set RUSTBGPD_ADDR on the container if configuration moves it. HTTP /livez is also available on the metrics listener; /readyz retains core-actor readiness checks.

  • Logs: [global.telemetry] log_format = "json" emits structured JSON. Bound retention when creating the container, for example with --log-driver=json-file --log-opt max-size=10m --log-opt max-file=3 on docker run. Add these flags to the container systemd unit's docker run command too. For Compose, set logging.driver: json-file and logging.options: {max-size: "10m", max-file: "3"} on the service. These limits keep approximately 30 MB per container; recreate existing containers to apply them. The image cannot configure the host logging driver. Forward logs to an aggregator when longer retention is needed. Successful read authorization audits are DEBUG; counters and existing denied, error, sensitive-read, and mutation audit levels are unchanged.

  • Networking: Linux FIB integration and BFD require access to the network namespace they operate on. For host addresses, use the root host-network path above; if the configuration also programs the host FIB or interfaces, add its documented NET_ADMIN opt-in. A bridge container affects only its own namespace. The interop suite runs rustbgpd as a containerlab kind: linux node, which is the cleanest reference setup for an isolated lab namespace.

Kubernetes probes

Kubernetes does not run the image's Docker health check; configure probes against the telemetry HTTP listener instead. The probes exist only when prometheus_addr is set under [global.telemetry] (the edge profile binds 0.0.0.0:9179). Their status codes and bodies are listed under HTTP probes.

containers:
  - name: rustbgpd
    image: ghcr.io/lance0/rustbgpd:latest   # pin a release tag in production
    ports:
      - name: telemetry
        containerPort: 9179
    startupProbe:
      httpGet: {path: /livez, port: telemetry}
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 60   # 10 minutes, the unit's TimeoutStartSec
    readinessProbe:
      httpGet: {path: /readyz, port: telemetry}
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3
    livenessProbe:
      httpGet: {path: /livez, port: telemetry}
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3
      terminationGracePeriodSeconds: 90   # liveness kills only; Kubernetes 1.25+
terminationGracePeriodSeconds: 1920   # ordinary stops; same as docker --stop-timeout

Readiness uses /readyz with the Kubernetes defaults: a 10 s period, a 1 s timeout and three failures. /readyz answers 200 ready when the PeerManager and RIB actors both respond within one 200 ms deadline and the gRPC server is not in the middle of an in-place restart. It means the control plane is up and admitting work. It does not mean the daemon has converged, received any routes, or established any peers; a daemon with zero Established peers can be ready.

That is deliberate for a route server or reflector. Peers open BGP sessions to a configured speaker address, not through a load-balanced Service, so readiness does not steer them toward a healthy replica. If readiness waited for Established peers, an outage on the peer side would mark every rustbgpd pod unready at once: the failure would appear to be in rustbgpd, rolling updates would stall, and restarting the pods could not fix it. Watch session state through its own signals instead, such as the BgpSessionNotEstablished rule in examples/prometheus/rustbgpd-alerts.yml.

Liveness uses /livez. It returns 200 ok whenever the HTTP listener accepts a connection and checks no actors. The listener has its own thread, so /livez keeps answering while the daemon's runtime workers are saturated. It restarts a process that is gone or cannot serve HTTP at all. It does not restart a daemon whose RIB or PeerManager has deadlocked while the listener still answers; Kubernetes has no equivalent of the systemd watchdog's actor check (see systemd). Alert on that case instead: a /readyz that stays 503, or bgp_readiness_probe_failures_total{reason=~"(peer_manager|rib)_(timeout|unavailable)"} still rising after several minutes, means the actors are not answering.

The daemon restarts the telemetry listener's HTTP server in place if that server crashes, instead of exiting (see Metrics/readiness server exits unexpectedly). A restart gap is at most 10 seconds, under the three consecutive failures this probe needs, so a server that comes back does not get the pod killed. A server that can never restart still fails liveness, and Kubernetes then restarts a daemon whose BGP side was healthy. To let BGP outlive that, point liveness at something other than this listener or raise failureThreshold.

The gRPC server is restarted in place the same way. While it is down /readyz answers 503 not ready: gRPC server is restarting, so the pod is unready for the length of the restart; /livez is not affected.

Do not point liveness at /readyz, with any failureThreshold. /readyz fails for conditions that are not a dead daemon:

  • The RIB reports an export-policy transition, selection-deferral release or peer teardown as stalled once it has run for 30 seconds, and keeps reporting it until the work finishes. The actor is answering and making progress the whole time. On a large table that work can outlast any threshold chosen in advance; a restart then drops every session and the daemon repeats the same work after it comes back.
  • Either core actor can miss the 200 ms deadline under heavy load and recover on its own.

The Kubernetes documentation warns that liveness probes that fail under load cause restarts that add to the load and can cascade. A restart of a BGP speaker resets every session it holds, so a liveness probe must fail only when a restart is the fix.

Startup. The listener is bound early, but nothing answers HTTP, not even /livez, until startup wiring has finished and the configured peer roster is installed, the same boundary as systemd READY=1. Until then, probes connect and time out. The startup probe uses /livez for that reason: its first 200 already means startup finished, and unlike /readyz it cannot be held red by a long RIB unit of work during initial table loading. It holds liveness off for the 10 minutes the systemd unit allows (TimeoutStartSec=10min). BGP ingress opens at that boundary, so routes from peers, including full tables, load while the probes already answer; the startup probe does not need to cover table loading.

Grace periods. Set the pod-level terminationGracePeriodSeconds to at least the stop budget described under systemd: an owned runtime-config mutation can take up to its 30-minute settlement watchdog plus a five-second grace to finish. During coordinated shutdown /readyz returns 503 not ready: daemon is shutting down.

A failed liveness probe only starts that same termination. Without a probe-level override, Kubernetes applies the pod-level grace period to a liveness kill too, so the time to restart a daemon that stopped serving HTTP is the detection time plus up to 32 minutes of shutdown. A wedged daemon usually cannot shut down cleanly: coordinated shutdown waits without a deadline for the PeerManager to drain its peers, and only a second termination signal skips that wait. Kubernetes sends one SIGTERM, then SIGKILL when the grace period ends.

The snippet therefore sets a probe-level terminationGracePeriodSeconds of 90 seconds on the liveness probe. The Kubernetes documentation supports the field from 1.25 and marks it stable since 1.28. The shutdown stages that have deadlines add up to about 70 seconds: a five-second runtime-config drain, then the pre-Cease ceilings of 30 seconds for the warm checkpoint, 5 seconds for the restart marker, 10 seconds for the EVPN runtime-apply fence and 5 seconds for each of four EVPN actor drains. 90 seconds covers them with headroom if the actors recover.

The 90 seconds is an intentional cap, not an estimate of a clean stop. A daemon that is still wedged when it expires is killed, and the PeerManager drain, which has no deadline, is deliberately not waited for. Graceful drain is moot by then: the process could not answer a plain HTTP request. The systemd path is no gentler: the shipped unit keeps systemd's default watchdog signal, SIGABRT, which the daemon does not handle, so a watchdog expiry ends the process without coordinated shutdown. In either case a runtime-config mutation that was cut short is reverted or adopted from its durable state on the next start (see settlement-watchdog.md). On clusters older than 1.25, the pod-level value applies to liveness kills as well.

Containerlab quick start

Containerlab is the easiest way to get a working rustbgpd ↔ FRR session on your laptop with no real routers. The simplest topology in the repo is tests/interop/m0-frr.clab.yml (abridged below; the file adds the config and start-script binds and the interface addresses):

name: m0-frr
topology:
  nodes:
    rustbgpd:
      kind: linux
      image: rustbgpd:dev
      cmd: sleep infinity
    frr:
      kind: linux
      image: quay.io/frrouting/frr:10.7.1
  links:
    - endpoints: ["rustbgpd:eth1", "frr:eth1"]

Bring it up:

# Build the dev image
docker build --target dev -t rustbgpd:dev .

# Deploy
sudo containerlab deploy -t tests/interop/m0-frr.clab.yml

# Start rustbgpd (the node runs `sleep infinity` until you do)
sudo docker exec -d clab-m0-frr-rustbgpd /usr/local/bin/start-rustbgpd.sh

# Check the session from the FRR side
sudo docker exec clab-m0-frr-frr vtysh -c "show bgp summary"

# More checks: the test driver scripts under tests/interop/scripts/ show
# the typical incantations for FRR vtysh + rbgp gRPC.

# Tear down
sudo containerlab destroy -t tests/interop/m0-frr.clab.yml --cleanup

For more topologies (route-reflector, EVPN, FlowSpec, BFD, etc.), see INTEROP.md.

The smallest deployment that exercises the full operator loop — build → validate → reload → observe — without depending on more than one peer. This is your "did I install this correctly" gate before scaling out.

  1. One rustbgpd instance + one FRR peer (or any compliant BGP peer). Provision the peer first; record its address + AS.

  2. Prometheus scrape on :9179. Add the scrape target in your Prometheus config:

    - job_name: rustbgpd
      static_configs:
        - targets: ["10.0.0.1:9179"]
  3. Config validation. Build the config, then:

    rustbgpd --check /etc/rustbgpd/config.toml

    Errors are rustc-style with file + line + carets; expect config OK on success. Run this every time before swapping the live config.

    A check that finds nothing to flag prints config OK. A check that flags something still exits 0, but the summary reads config VALID, <n> WARNING(S) — NOT a clean check and the warnings are framed on stderr above it. The warnings identify: a configured eBGP neighbor or dynamic range with no explicit policy in a direction (unfiltered when [global] ebgp_requires_policy is off, carrying no routes in that direction when it is on); an RFC 8212 posture still inherited from legacy omission (rfc8212_secure_default_ready — set an explicit root config_epoch plus [global] ebgp_requires_policy); an orr_vantage with no linkstate-negotiating neighbor; and, when a gRPC TCP listener sets a positive tls_expiry_warning_seconds, a TLS certificate whose notAfter falls inside that window. None is rejected — a permit-all route server is a legitimate configuration — but none should reach production unnoticed.

  4. First start.

    sudo systemctl start rustbgpd
    journalctl -u rustbgpd -f

    Watch for the session to reach Established:

    rbgp neighbor
  5. Edit + reload cycle. Edit a copy of the config, then dry-run the diff of the candidate against the current on-disk config (--diff takes the candidate; the positional argument is the config to compare against, default /etc/rustbgpd/config.toml):

    cp /etc/rustbgpd/config.toml /tmp/new-config.toml
    $EDITOR /tmp/new-config.toml
    rustbgpd --diff /tmp/new-config.toml /etc/rustbgpd/config.toml

    When the diff looks right, install the candidate over /etc/rustbgpd/config.toml before reloading.

    --diff calls into the same ConfigDiff machinery the reload path uses; what it reports is what reload will do. Cross-reference against the reload matrix to confirm which changes hot-apply and which are restart-required. Apply:

    sudo systemctl reload rustbgpd
  6. Restart cycle. Confirm the FRR peer reconverges within the GR timer (GR is on by default):

    sudo systemctl restart rustbgpd
    # FRR should retain the routes during the restart window; rustbgpd
    # advertises R=1 in the GR capability on the next OPEN.
  7. Observability sanity-check. Read at least one Prometheus counter and one neighbor field:

    curl -s http://10.0.0.1:9179/metrics \
      | grep -E "bgp_session_established_total|bgp_messages_received_total"
    rbgp neighbor 10.0.0.2

If all six steps work end-to-end, your install is sound. Scale from there: add more peers, add policy chains, wire BMP / gNMI / MRT according to your needs.

Config validation workflow

These cover the config lifecycle, from bootstrap to reload:

CommandWhat it does
rustbgpd --init-config <lab|edge|route-server>Print a curated, commented starter TOML to stdout and exit (file output is not yet supported). lab is a minimal single-box profile; edge is an eBGP edge skeleton; route-server is a self-contained IPv4 two-member skeleton with fail-closed import policy and transparent export. Every profile uses a mode-0600 local UDS whose filesystem permissions authenticate access as the implicit local-operator, sets [global] ebgp_requires_policy = true, and passes --check --strict as emitted. Cannot be combined with --check / --diff.
rustbgpd --check <file>Parse + validate; print config OK, config VALID, <n> WARNINGS — NOT a clean check (warnings framed on stderr; still exit 0), or a rustc-style diagnostic (exit 1). Does not start the daemon. The running daemon logs the same config advisories as WARN lines: all of them at startup, and after a SIGHUP reload or API config change is adopted, each advisory that the previous config did not raise.
rustbgpd --check --strict <file>The same check, but any warning exits 1 instead of 0 — for CI and deployment gates that must not accept a valid-but-risky config. A clean check still exits 0. --strict without --check is an error (exit 2).
rbgp policy check [--fail-on-empty] [--deny-lints[=LINT,...]] [-j] <file.rpol>Compile a .rpol file and run its test blocks offline, without a daemon — a CI or editor gate for policy files. Exit 0 = clean, 1 = compile error, 2 = failing test (or no tests under --fail-on-empty), 3 = coverage below a requested threshold, 4 = a lint denied by --deny-lints. -j prints one rbgp-policy-check/1.0 document with located findings. See policy check as a CI and editor gate.
rustbgpd --migrate-config <pin-legacy|prepare-secure> --offline [--dry-run] <file>Rewrite the RFC 8212 posture representation in place, offline (Linux only). pin-legacy writes epoch 1 + explicit false. prepare-secure writes epoch 2 + explicit true. --dry-run performs the same validation proof and discards the stage. --offline is an operator assertion, not a daemon probe — stop or quiesce the daemon first.
rustbgpd --diff <candidate> [<current>]Compare a candidate file against the current on-disk config (<current> defaults to /etc/rustbgpd/config.toml); print per-section change list with expected reload class and the SIGHUP reload route. Exit 0 = no actionable changes, 1 = actionable changes, 2 = load error. This is a static file-vs-file compare: once the candidate is installed over <current>, it reports no changes. To compare against the running daemon's view, use rbgp config diff.
rbgp config diff <candidate>Compare a candidate file against the running daemon's config snapshot, with the same reload-class annotations. Run it on the installed file after a reload to see what did not take effect, such as restart-required edits. Exit 0 = no changes, 2 = changes present, 1 = error. The 1 and 2 codes are the reverse of rustbgpd --diff.
systemctl reload rustbgpd (or kill -HUP $(pidof rustbgpd))Apply the diff. Live fields hot-apply; restart-required fields are pinned and logged at ERROR (the live values are kept).

The validation pipeline is the same in all three places: TOML parse → validate() → ConfigDiff. A config that passes --check will not error on --diff; a --diff that loads cleanly and does not report SIGHUP reload route: rejected will not error on reload. A rejected route means the reload would change nothing; split it as the reason says (see SIGHUP reload routes).

Observability

Prometheus

The exporter binds at [global.telemetry] prometheus_addr. The same listener serves /livez and /readyz for orchestrators. Key counters operators watch:

  • Session liveness — bgp_session_established_total, bgp_session_flaps_total, bgp_messages_received_total, bgp_messages_sent_total (all by peer).
  • Loop / leak detection — bgp_as_path_loop_detected_total, bgp_rr_loop_detected_total, bgp_otc_routes_blocked_total{peer, reason} (RFC 9234 / ADR-0071), bgp_role_mismatch_total{peer, local_role, remote_role}. Canonical reason values are documented in docs/reference/operations.md ("Ingress rejection / route-leak detection").
  • Route processing — bgp_rib_prefixes{peer, family}, bgp_rib_loc_prefixes{family}, bgp_max_prefix_exceeded_total.
  • GR / FIB — bgp_gr_active_peers, bgp_gr_stale_routes, bgp_fib_routes_installed_total, bgp_fib_kernel_failures_total.
  • Allocator — jemalloc_allocated_bytes, jemalloc_active_bytes, jemalloc_resident_bytes, jemalloc_mapped_bytes. Present in builds with the jemalloc feature (the published container image and release tarballs); refreshed at scrape time. allocated is live application bytes, resident is jemalloc's contribution to RSS — a widening gap between the two is retained-but-unused allocator memory, the first thing to check before suspecting a leak. This jemalloc build takes run-time options from _RJEM_MALLOC_CONF, not MALLOC_CONF; see heap profiling with jemalloc.

Policy filtering visibility — Prometheus. bgp_policy_routes_total {peer, policy, direction, action} attributes each import and export policy evaluation to the decisive member on Deny. A nonempty chain that completes without rejection uses the bounded policy="chain_default_permit" sentinel; policy="inline" remains the label for an inline Deny or a Permit from an absent / genuinely empty chain. Initial table dumps, route refreshes, dirty resyncs, and forced outbound refreshes can increment export counters because they re-evaluate export policy. Each label identity remains monotonic while its policy/action is reachable from the installed chain. After a successful policy replacement, an unreachable identity is retired and goes stale; reinstalling that exact identity creates a fresh counter starting from zero. Prometheus rate() and increase() handle the resulting counter reset.

# Routes denied by a named filter on each peer's import side:
rate(bgp_policy_routes_total{direction="import", action="deny"}[5m])

An operator policy literally named chain_default_permit remains ordinary Deny attribution. The sentinel interpretation applies only when action="permit".

Policy evaluation errors — Prometheus. bgp_policy_eval_errors_total {direction, kind} counts routes denied by the fail-closed evaluation- error rail (ADR-0103 Decision 4: an action that needs an absent operand, fuel/loop-cap exhaustion, ...; arithmetic saturates and cannot fail). These denies also appear in bgp_policy_routes_total as action="deny"; this counter separates "the policy said no" from "the policy is broken". The exception is direction="condition": a conditional advertisement's condition_policy error denies no route and does not appear in bgp_policy_routes_total; it can leave the condition unknown. Any nonzero rate deserves a look. For import and export errors, rbgp policy stats --direction both names the failing chain, policy, and term (eval_errors count + last_error per chain), and the rate-limited daemon WARN carries the same blame line. For condition errors, bgp_conditional_advertisement_condition{state="unknown"} names the definition, and rbgp rib --prefix P advertised PEER --explain for a route its advertise_policy selects names the failing condition_policy and term in the conditional_advertisement step.

# Any policy erroring anywhere is alert-worthy:
sum by (direction, kind) (rate(bgp_policy_eval_errors_total[5m])) > 0

Policy filtering visibility — gRPC scalar aggregates. NeighborState carries four per-peer running totals to give operators a cheap sanity-check on the labeled Prometheus counter:

FieldDirectionScope
import_policy_routes_permittedimportDaemon lifetime per neighbor — survives session resets and reconfigures; cleared by daemon restart or neighbor deletion.
import_policy_routes_deniedimportDaemon lifetime per neighbor — same.
export_policy_routes_permittedexportDaemon lifetime per neighbor — same.
export_policy_routes_deniedexportDaemon lifetime per neighbor — same.

Both directions keep counting across reconnects, like updates_* and flap_count. For a rate or a per-session window, use increase() over bgp_policy_routes_total. The CLI surfaces these in rbgp neighbor <peer> as a Policy Stats block:

Policy Stats:
  Import:                 permitted=1247 denied=31
  Export:                 permitted=892 denied=0

JSON output (rbgp --json neighbor <peer>) carries the same fields under import_policy_routes_permitted / import_policy_routes_denied / export_policy_routes_permitted / export_policy_routes_denied, elided when zero.

BMP

If [bmp] is configured, rustbgpd opens a TCP session to the BMP collector and exports per-peer Adj-RIB-In + Peer Up / Peer Down events. See CONFIGURATION.md for the schema.

gNMI

The gNMI adapter (ADR-0070) exposes Capabilities / Get / Subscribe telemetry plus the static numbered-neighbor Set subset over the same sockets as the native API. Native gNMI is registered on [global.telemetry.grpc_tcp] only when native mTLS is configured; the [global.telemetry.grpc_uds] listener serves gNMI unconditionally as a local-only extension. RFC 7951 JSON encoding. See GNMI.md for the path namespace and supported mutation leaves.

CLI introspection

rbgp is the operator interface for read queries.

rbgp neighbor                  # list all neighbors
rbgp neighbor 10.0.0.2         # detail
rbgp neighbor 10.0.0.2 --compare 10.0.0.3  # live update-group relationship
rbgp rib                       # browse Loc-RIB
rbgp bfd                       # BFD sessions (ADR-0067)
rbgp evpn                      # EVPN instances + Type 2/3 RIB
rbgp top                       # live TUI dashboard

Most data-oriented read commands support --json for scripting. Commands with fixed formats, such as metrics, completions, and top, keep their command-specific output.

For an Established neighbor, rbgp neighbor <addr> reports whether the peer negotiated Route Refresh, Enhanced Route Refresh, and Extended Messages, plus the directional maximum BGP message size rustbgpd may send (4096 or 65535 bytes). The JSON fields preserve false as an explicit negotiated result and omit values an older daemon did not expose; a missing live-session snapshot is reported separately by negotiation_available.

Package upgrade and rollback

This is the canonical procedure for the release .deb and .rpm packages. The package scripts create the service user and run systemctl daemon-reload; on upgrade they deliberately do not stop, start, or restart rustbgpd. An operator must therefore own the graceful stop and the verification that follows it. An upgrade over a running daemon leaves the old binary running, and the package says a restart is required; rbgp health reports the version of the running daemon, not the file on disk.

Upgrade

  1. Read the target release's CHANGELOG section and verify the downloaded package against its release checksum. Config compatibility is not assumed across minor releases.

  2. Extract the package into a temporary directory and run the new binary's strict check against the configuration that the service will use:

    candidate_pkg=/path/to/rustbgpd_X.Y.Z_amd64.deb # or the .rpm
    candidate_root=$(mktemp -d)
    chmod 0755 "$candidate_root"
    
    case "$candidate_pkg" in
      *.deb) dpkg-deb --extract "$candidate_pkg" "$candidate_root" ;;
      *.rpm) rpm2cpio "$candidate_pkg" |
               (cd "$candidate_root" && cpio --quiet -id --no-absolute-filenames) ;;
      *) echo "expected a .deb or .rpm" >&2; exit 2 ;;
    esac
    
    sudo -u rustbgpd \
      "$candidate_root/usr/bin/rustbgpd" --check --strict \
      /etc/rustbgpd/config.toml

    Fix every error before proceeding. There is no general schema-migration tool; make release-note field renames by hand, then repeat the candidate check. The one exception is the RFC 8212 posture representation, which rustbgpd --migrate-config pin-legacy|prepare-secure --offline [--dry-run] CONFIG_PATH rewrites in place (see CONFIGURATION.md → config_epoch); stop or quiesce the daemon first — --offline is an operator assertion, not a daemon probe. This check validates candidate config bytes only: it does not inspect runtime_state_dir or config-adjacent commit-confirm authority.

    For the import explain and rejected-route retention capacity limit, older configs with [policy.explain] cache_size or [policy.reject_retention] capacity above 2,097,152 now fail startup and reload validation, even if that section has enabled = false. Reduce each named value to at most 2,097,152, then run the new binary's --check --strict against the edited file before stopping or restarting the daemon. The defaults remain 4096 and 1024; zero still acts as one.

    Before any upgrade, use the still-running daemon to run the pre-upgrade diagnostic against the file the new binary will boot:

    sudo -u rustbgpd "$candidate_root/usr/bin/rbgp" doctor --pre-upgrade /etc/rustbgpd/config.toml

    It is red, with the next action, while a confirmed transaction is pending, applying, rollback-failed, or ambiguous (confirm or abort it with rbgp config confirm <id> / rbgp config abort <id>), while a runtime-config settlement owner is still settling, when the file resolves to a different RFC 8212 epoch/posture than the live daemon runs, and whenever the evidence is unavailable or denied. It never confirms, aborts, rewrites, or stops anything. A green result is an observation at one instant (human output says Pre-upgrade observation as of unix <t>; JSON carries observed_at_unix_seconds), not a fence: a transaction can still start after it, which is why step 3 stops the daemon and then repeats these checks. See the check reference.

    When the installed release is v0.64.0 or earlier, also clear retired commit-confirm authority first: a locator-free commit-confirm-journal.json or a retired v2 *.commit-confirm-locator.json must be recovered with exactly rustbgpd v0.64.0. Do not proceed while either artifact is present or inaccessible; delete one only after proving the transaction terminal and the current config intended. v0.65.0 and every later release otherwise refuse boot and leave the authority untouched.

  3. Stop the running daemon cleanly, then confirm it is down:

    sudo systemctl stop rustbgpd
    systemctl is-active rustbgpd # expected: inactive

    The coordinated stop writes the GR restart marker. With GR enabled, the new process can advertise R=1 while sessions rebuild. Families with configured kernel installers (FIB, blackhole discard, EVPN) still advertise forwarding_preserved = false, while control-plane-only families advertise F=1 under the role rules. This is not a forwarding-continuity guarantee; use a drained route-server pair when traffic continuity matters.

    If a controller injects routes that must survive the restart at a GR helper (for example blackholes), consider gr_eor_grace_time on that neighbor: End-of-RIB then waits for the controller's replay, up to a fixed deadline, instead of leaving with the initial table.

    With the daemon inactive, repeat the checks that only a stopped daemon can make authoritative: verify the absence of a config-adjacent *.commit-confirm-locator.json, make any explicit offline posture correction the check called for, then repeat the candidate --check --strict from step 2 against the final file after every rewrite. The live diagnostic observed an earlier instant; these post-stop checks are the ones that hold.

  4. Install the already-checked package:

    sudo apt-get install -y "$candidate_pkg" # Debian / Ubuntu
    sudo dnf install -y "$candidate_pkg"     # RHEL / Rocky / Alma

    Use the command for the host's package manager, not both. The package marks /etc/rustbgpd/config.toml as config|noreplace: a locally modified config stays in place. Inspect any replacement candidate reported by the package manager before starting (normally .dpkg-dist or .rpmnew):

    sudo find /etc/rustbgpd -maxdepth 1 -type f \
      \( -name '*.dpkg-dist' -o -name '*.rpmnew' \) -print
    sudo -u rustbgpd /usr/bin/rustbgpd --check --strict \
      /etc/rustbgpd/config.toml
  5. Start and verify the installed version and its operator surfaces:

    sudo systemctl start rustbgpd
    sudo systemctl --no-pager --full status rustbgpd
    dpkg-query -W -f='${Version}\n' rustbgpd # Debian / Ubuntu
    rpm -q --qf '%{VERSION}\n' rustbgpd     # RHEL / Rocky / Alma
    sudo -u rustbgpd rbgp health
    sudo -u rustbgpd rbgp summary

    Confirm that the Version: line from rbgp health, which the running daemon reports, matches the package you selected. rustbgpd --version reads the binary on disk, which an unrestarted daemon no longer runs. Wait for every expected session to return to Established; systemd active alone does not prove routing convergence.

    If the general FIB installer has a v1–v4 fib-owned.json, start a release with the verified v5 rewrite using the existing FIB configuration and runtime state directory before upgrading to one that requires v5. The v5 writer shipped in v0.29.0, but later releases could leave older receipts untouched when routes stayed the same. After startup, check that fib-owned.json in the configured runtime_state_dir has "version": 5. Resolve any write or verification error in the daemon log before continuing.

Rollback

A rollback is another compatibility migration, not merely a package command. Read the target version's release notes. Extract the old package as above and run its rustbgpd --check --strict against the intended rollback config before stopping the current daemon. If the old binary rejects the current config, restore a version-controlled config known to that release, preflight that exact file with the old binary, and install it only after the stop.

Canonical persistence omits genuinely optional sections while they remain at their defaults. This keeps routine runtime changes from adding unused feature tables to the file. Omitted values use the receiving release's defaults, as they do in hand-written TOML; this is not a cross-release semantic round-trip or downgrade compatibility promise. Once a new feature is configured, its section is retained; renamed fields use the current canonical spelling. An older binary can therefore still reject a newer persisted config. Restoring an older version-controlled config also discards API-driven changes made since that saved copy, so reconcile those changes explicitly before the rollback.

Before downgrading, run rbgp doctor --pre-upgrade against the intended rollback config (or rbgp config status) and finish a pending confirmed transaction with rbgp config confirm <id> or rbgp config abort <id>. Do not delete a live locator to force the downgrade. Once the transaction is terminal, verify that the config-adjacent *.commit-confirm-locator.json and the retired locator-free commit-confirm-journal.json are absent. The fixed v3 raw prior and metadata may remain after a warning-only cleanup failure: once locator absence is verified they are non-authoritative, and rustbgpd v0.64.0 ignores those commit-confirm v3-only names. If the target predates either retained history format, preserve and move the complete config-history/ directory aside after stopping the daemon and before installing the older binary. In particular, writable downgrade after a config-history v3 row has been published is unsupported without moving the directory aside: the older writer ignores v3, can expose stale rows, and may reuse its sequence. There is no backfill or down-conversion. Re-upgrade fails closed on duplicate sequences or an over-cap roster; it cannot make an old writer's collisions trustworthy. Builds that recognize newer-format history rows (see ADR-0124) list them as unreadable and skip recording, with a warning, until they are moved aside.

For a rollback to v0.64.0 that needs an epoch-aware config converted, v0.83.0 was the last release with the Linux downgrade-v0.64 action. Extract both release binaries. After settling pending transactions and stopping the daemon, run the v0.83.0 binary on a copy of the config alongside the original so relative paths resolve from the same directory:

/path/to/v0.83.0/rustbgpd --migrate-config downgrade-v0.64 --offline \
  --validator /path/to/v0.64.0/rustbgpd /etc/rustbgpd/rollback.toml

This converts only the RFC 8212 posture and preserves its effective boolean. If either release rejects the config, use a saved v0.64.0 config and reconcile later changes. Run the v0.64.0 binary's --check --strict on the rollback file before installing it.

For the package rollback, stop the daemon, move history aside if required above, and install the selected older package:

previous_pkg=/path/to/rustbgpd_W.X.Y_amd64.deb # or the .rpm
sudo systemctl stop rustbgpd
# Preserve and move config-history/ aside here if the target cannot read it.
sudo apt-get install -y --allow-downgrades "$previous_pkg" # Debian / Ubuntu
sudo dnf install -y "$previous_pkg"                       # RHEL / Rocky / Alma

Inspect any .dpkg-dist/.rpmnew, install the preflighted compatible config, then repeat the start and verification commands from the upgrade procedure. A binary that predates GR marker v3 rejects the v3 marker and cold-starts without restarting-speaker mode. Do not count on peer route retention across that boundary; drain the route-server member of a redundant pair first when continuity matters.

Persistent state on disk

Runtime state lives in runtime_state_dir (default /var/lib/rustbgpd). The v3 commit-confirm locator instead lives beside the lexical launch config so startup finds the sole pending authority before trusting candidate contents:

Assign a distinct runtime_state_dir to every concurrently running rustbgpd daemon. The directory is single-writer runtime state; sharing it between live processes is unsupported even when their configuration files differ.

PathPurposeSurvives restart
gr-restart.tomlGraceful Restart coordination marker. Written on clean shutdown, read on startup to set the R-bit in OPEN.Yes
warm-bundle-v1/Optional owner-private shutdown checkpoint (manifest.json plus a content-addressed MRT artifact). Published only when warm_cache_checkpoint_on_shutdown = true; not restored on startup.Yes
<runtime_state_dir>/commit-confirm-journal.jsonRetired locator-free v1/v2 authority; secret-bearing normalized TOML. Its presence makes v0.65.0 and every later release refuse boot untouched.Until recovery with rustbgpd v0.64.0 or operator-proven terminal deletion
<runtime_state_dir>/commit-confirm-v3-prior.tomlFixed v3 raw accepted prior; secret-bearing normalized TOML. Published first.Until terminal cleanup; safe residue may remain
<runtime_state_dir>/commit-confirm-v3-metadata.jsonFixed confidential provenance, digest, and file-identity metadata; no raw TOML. Published after the raw prior.Until terminal cleanup; safe residue may remain
<absolute lexical config path>.commit-confirm-locator.jsonConfidential paths/digests and sole v3 pending boot authority; no raw TOML. Published last and checked before candidate contents.Until durable unlink and parent sync make the transaction terminal
config-history/*Last 20 owner-private v2/v3 JSON rows for rbgp config history; newest is index 0. V2 retains secret-bearing TOML up to 10 MiB; larger accepted snapshots receive at-most-64-KiB metadata-only v3 rows without payloads or source rosters. V3 is permanently rollback-ineligible and independent of commit-confirm recovery. Retired TOML files are ignored and retained. V2 records hash but do not archive external sources and are rollback-eligible only when live sources exactly reproduce recorded provenance.Yes
fib-owned.jsonFIB ownership receipt — which kernel routes the daemon installed (ADR-0061). Used to drain orphan installs on next start.Yes
fib-owned.json.staleEvidence set aside from a FIB ownership receipt at startup. An unsupported (newer) version or a payload shape that contradicts its version renames the receipt here, and the daemon starts owning nothing. A changed [[fib_tables]] signature copies the receipt here instead and keeps the live fib-owned.json, so unchanged tables keep their owned routes. Diagnostic only; never read back.Yes; replaced by the next one
blackhole-owned.jsonExact BLACKHOLE prefix authority. Mode 0600; adoption and deletion require this receipt plus the kernel marker.Yes
events.db, events.db-wal, events.db-shm, events.last_id, events.db.stale*Left behind by the removed durable event history. Current releases never open them; delete them after upgrading. Settle any pending commit-confirm with the previous release first: a retained prior that still holds [event_history] stops the upgraded daemon at startup (see the removal note).Not used
crash/panic-*.tomlPanic reports (message, location, thread, binary version, timestamp; never environment or arguments), written through a temporary file and renamed into place. Write-only; the newest 10 are kept, and rbgp doctor includes them in support bundles. An interrupted write can leave a panic-*.toml.tmp file, which retention and rbgp doctor ignore.Yes
grpc.sockgRPC UDS endpoint (if [global.telemetry.grpc_uds] configured).Recreated on start

Stop the daemon before deleting blackhole-owned.json; deletion intentionally preserves all surviving marker rows as foreign. Never share runtime_state_dir between live instances, and do not downgrade to marker-only adoption without a drain plan.

The lexical config, metadata, and raw-prior paths resolve to absolute identities. A writer or present pending object requires daemon-owned real parents that are not group- or world-writable; locator absence carries no authority or ordinary-startup storage requirement. Pending and staging files must be daemon-owned regular files with mode 0600; symlinks and special files fail closed. Locator unlink plus parent fsync is terminal; only subsequent verified metadata/raw cleanup and pending-directory fsync are warning-only.

Production reads and writes v3. Retired v1/v2 authority refuses boot untouched; finish it before upgrade or recover with rustbgpd v0.64.0. Offline --check does not inspect this runtime state. Before downgrade, finish or abort v3 and never delete its live locator manually.

Routing state is not restored. The optional shutdown checkpoint contains only eligible post-import-policy Adj-RIB-In views for future use; Loc-RIB, Adj-RIB-Out, and policy evaluation state are not checkpointed, and the current startup path loads none of it. Routing state rebuilds from peer routes after restart. GR and checkpoint state support bounded control-plane restart handling, but do not guarantee forwarding continuity; peers may withdraw or replace routes, and therefore forwarding, until normal convergence.

The config file itself is mutable across runs: neighbor add/delete operations via gRPC persist back to the config file (see CONFIGURATION.md → "Config Persistence").

Migrate whole-table family reads

Before the v1 inventory freeze, the scoped RR-only read contract moves to bounded pages. Replace ListVpnRoutes, ListLabeledRoutes and ListRtcRoutes with ListVpnRibRoutes, ListLabeledRibRoutes and ListRtcRibRoutes, selecting view = BEST for the previous winner-only behavior. Replace ListBgpLsRoutes with ListBgpLsRibRoutes and view = BEST; its old peer_filter becomes source_peer_filter. Replace ListTopologyNodes and ListTopologyLinks with ListOrrTopologyNodes and ListOrrTopologyLinks. Replace ListEvpnRoutes with ListBestEvpnRoutes; select each of Types 1–5 separately for the scoped EVPN role. The seven whole-table methods remain functional but are deprecated and outside v1. No RPC is removed. Plain CLI JSON arrays keep their shape; rbgp builds them from the bounded pages, in the pages' key order.

Follow next_page_token until empty. A table mutation returns ABORTED; restart the walk with an empty token. Page size 0 defaults to 100; values above 1000 clamp to 1000. Older daemons can return UNIMPLEMENTED for the bounded methods, so check server support before switching automation. See the family page contract for view, filter and peer semantics. Clients using additive EVPN peer filters must also verify the documented response acknowledgement before using rows.

Sample profiles

The repo ships eleven config profiles, plus a Docker Compose quick-start, under examples/ covering the standard deployment shapes. Pick the closest match, copy, edit:

ProfileFileUse case
Minimalexamples/minimal/config.tomlSingle eBGP peer; dev-friendly, state in /tmp.
IX route serverexamples/route-server/config.tomlRPKI, Add-Path, dual-stack, per-member policy chains.
Route reflectorexamples/route-reflector/config.tomlIPv4/IPv6 unicast reflector: client peer group, dynamic client range, non-client peer, Add-Path send, GR/LLGR.
MANRS IXP Action 1examples/manrs-action1/config.tomlRoute server with RPKI-invalid rejection and IRR-derived member filtering; walkthrough in cookbook/manrs-ixp-action1.md.
Fabric edge / Linux FIBexamples/linux-edge-fib/config.tomlFIB integration on configured unicast tables; ECMP, weighted multipath.
EVPN VTEP leafexamples/evpn-vtep-leaf/config.tomlBidirectional VTEP: kernel FDB → Type 2 origination.
EVPN RR fabricexamples/rr-evpn-fabric/config.tomlRoute Reflector mode, stateless EVI (no [[evpn_instances]]).
Route collectorexamples/route-collector/config.tomlPassive listener, MRT dumps, BMP export.
FlowSpec controllerexamples/flowspec-controller/config.tomlgRPC-driven IPv4/IPv6 FlowSpec origination to border routers over FlowSpec-only iBGP sessions.
DDoS mitigationexamples/ddos-mitigation/config.tomlFlowSpec injection + RTBH (RFC 7999 BLACKHOLE).
Hosting providerexamples/hosting-provider/config.tomliBGP injector for customer-prefix automation.
Docker Composeexamples/docker-compose/Quick-start with an FRR peer (gRPC on TCP).

Each profile validates with rustbgpd --check out of the box; edit the AS, addresses, and TLS material before deploying.

Security checklist

See SECURITY.md for the full posture document. The short version for first deployment:

  • Bind addresses. prometheus_addr, grpc_tcp.address, grpc_uds.path default to listening on what the config says. Don't expose the Prometheus or gRPC endpoint to untrusted networks without authentication.
  • gRPC. Use mTLS in production. Set [security.grpc] enforcement = "tier" and map principals to roles under [security.grpc.roles] (observer / automation / operator) per SECURITY.md; the per-method tier matrix itself is compiled into the daemon (crates/api/src/authz.rs), not configured per-tier in TOML. UDS with a restrictive mode is fine for single-host operator access.
  • BGP authentication. TCP-MD5 (RFC 2385) and TCP-AO (RFC 5925) are both supported. TCP-AO is preferred and needs mainline Linux 6.7 or newer, or a downstream kernel with TCP-AO backported, built with CONFIG_TCP_AO=y; see ADR-0062.
  • RPKI cache transport. RTR (RFC 8210) sessions are TCP with optional per-cache md5_password or tcp_ao authentication; they are never encrypted. Set a key when the cache supports one; otherwise run caches on loopback or a trusted segment, or tunnel the session (WireGuard, an SSH or stunnel forward). A remote cache reached over an untrusted path is a validation-integrity problem: anything on the path can rewrite the VRPs the daemon validates against.
  • Firewall. Allow inbound TCP/179 only from configured peer addresses (or address ranges if using [[dynamic_neighbors]]). Block everything else.

Troubleshooting

SymptomWhere to look
Session won't establishrbgp neighbor <addr> + journalctl -u rustbgpd -p warning (or, from JSON logs, journalctl -u rustbgpd -o cat | jq -c 'select(.level=="WARN" or .level=="ERROR")')
Routes received but not installedrbgp rib, then rbgp rib received <peer> --rejected; for the statement-level trace, rbgp policy explain --neighbor <peer> --prefix <CIDR> --direction import (opt-in: needs [policy.explain] enabled = true)
Reload didn't change behaviorrbgp config last-reload for pending restart-required fields and the outcome (known_partial lists unapplied identities), then rbgp config diff /etc/rustbgpd/config.toml for edits the running config does not hold, such as restart-required fields; cross-reference the reload matrix. rustbgpd --diff compares two files, so it reports no changes once the new file is installed
FIB programming failuresbgp_fib_kernel_failures_total Prometheus counter + journalctl -u rustbgpd -g 'failed to apply general FIB route op'
FIB or BLACKHOLE planning stops before kernel accessincrease(bgp_dataplane_reconcile_planning_failures_total[10m]) > 0 + matching structured warning; status remains the last successful snapshot
EVPN-specific issuesevpn-vtep-troubleshooting.md
Anything not aboveOPERATIONS.md "Debugging" section

For deeper investigation, raise the daemon log level globally with the RUST_LOG environment variable — there is no log-level key in [global.telemetry]; log_format selects "json" or "text" output and does not control verbosity. Under systemd, set the log level in the unit's [Service] section:

[Service]
Environment=RUST_LOG=debug

RUST_LOG accepts the usual tracing filter syntax, so a targeted directive beats a daemon-wide debug — including the per-peer span form documented in OPERATIONS.md:

Environment=RUST_LOG=info,[peer{peer_addr=10.0.0.1}]=debug

Per-peer verbosity also comes from [[neighbors]] log_level = "debug".

When systemd connects the daemon's stdout to the journal (it sets JOURNAL_STREAM to stdout's device and inode), each line starts with its level's syslog priority, which journald strips and stores: ERROR as err (3), WARN as warning (4), INFO as info (6), and DEBUG/TRACE as debug (7). This detection is automatic; it needs no configuration and applies to both log formats. journalctl -p therefore filters by the daemon's level. Output to a terminal, a file or a pipe has no prefix. Restart required for the global setting; per-peer log_level is live — replaced on SIGHUP, so a level edit reaches running sessions without a restart (see the reload matrix).

Source on GitHub

On this page