Deploying rustbgpd

Install, operate, and upgrade rustbgpd.

This is the end-to-end operator runbook: install, run, validate, reload, observe, and upgrade. It's intentionally smaller than the M- series interop matrix in INTEROP.md — that document proves rustbgpd interops cleanly with FRR / BIRD / GoBGP, this one gets your daemon onto your box.

For deeper references:

• CONFIGURATION.md — every config field, with examples.
• reload-matrix.md — which fields hot-apply vs. need a restart.
• OPERATIONS.md — debugging, log filtering, failure modes.
• SECURITY.md — firewalling, gRPC mTLS, authorization tiers.

Status & expectations

rustbgpd is public alpha overall. The exception is the narrow, machine-pinned v1 route-server / route-reflector contract, which covers only its listed config fields, native gRPC signatures, CLI/JSON contracts, and control-plane roles. Unlisted TOML and API surfaces can still change between minor versions. See CHANGELOG.md for migration notes and run rustbgpd --check <new config> against the new binary before swapping it in.

The supported daemon targets are Linux x86_64 and aarch64. Published binaries use a glibc 2.31 baseline. See the canonical platform support contract for CI and non-Linux boundaries.

Suitable for: lab pilots, data-center fabric pilots, IX route-server pilots, automation-heavy control-plane work where the API surface evolution is acceptable. Not yet suitable for fully unattended production deployments where the operator can't react to a CHANGELOG note.

Install

Verified installer

The installer ships as the install.sh release asset from v0.70.0 onward (earlier releases do not include it) and is also available in a source checkout as packaging/install.sh. It resolves latest once, then downloads the selected artifact and its matching per-architecture checksum manifest from that same tag. It refuses unknown architectures, non-glibc hosts, glibc below 2.31, missing checksum rows, duplicate checksum rows, and checksum mismatches before installing.

On Debian/Ubuntu it installs the verified native .deb; on RHEL, Rocky, and AlmaLinux 9+ it installs the verified native .rpm. The package keeps the existing non-replacing configuration behavior. The installer never enables or starts a service; edit the configuration and choose that step yourself.

For a reproducible install from this checkout, inspect and run the installer with an explicit stable tag (vMAJOR.MINOR.PATCH):

less packaging/install.sh
sh packaging/install.sh --tag v0.75.0

Without a source checkout, fetch the published asset:

curl -fsSL https://github.com/lance0/rustbgpd/releases/latest/download/install.sh | sh

Other GNU/Linux distributions must use an explicit prefix or download-only mode. --prefix extracts the verified tarball's existing layout directly into an empty directory; it does not create users, write configuration, or install systemd units. --download-only writes the verified selected artifact and its manifest without installing either.

sh packaging/install.sh --tag v0.75.0 --prefix /opt/rustbgpd-0.75.0
/opt/rustbgpd-0.75.0/rbgp doctor

sh packaging/install.sh --tag v0.75.0 --download-only ./rustbgpd-v0.75.0

Pre-built binary tarball

Tagged releases publish rustbgpd-linux-amd64.tar.gz and rustbgpd-linux-arm64.tar.gz under GitHub Releases. Each ships rustbgpd (the daemon), rbgp (the CLI), rs-config-render (the route-server config renderer), and birdwatcher-adapter (the Alice-LG shim), plus man pages and shell completions under share/:

rustbgpd, rbgp, rs-config-render        binaries
birdwatcher-adapter                     Alice-LG birdwatcher shim
LICENSE-MIT, LICENSE-APACHE             licenses
rustbgpd.schema.json                    config JSON Schema
share/man/man1/rbgp.1                   CLI man page
share/man/man8/rustbgpd.8               daemon man page
share/completions/rbgp.{bash,zsh,fish}  shell completions
share/systemd/rustbgpd.service          hardened systemd unit
share/systemd/rustbgpd@.service         opt-in per-handle systemd unit
share/systemd/rustbgpd-dataplane.conf   opt-in CAP_NET_ADMIN drop-in
share/monitoring/rustbgpd-overview.json Grafana overview dashboard
share/monitoring/rustbgpd-evpn.json     Alpha EVPN Grafana dashboard
share/monitoring/rustbgpd-alerts.yml    Prometheus alert rules
share/monitoring/rustbgpd-alerts_test.yml promtool rule tests

The binaries are built against glibc 2.31; any distro at or above it works (Debian 11+, Ubuntu 22.04+, RHEL/Rocky/Alma 9+).

The filename is the same on every release; releases/latest/download/ always resolves to the current tag, so this snippet never needs a version bump.

# Pick the right arch.
SUFFIX=linux-amd64    # or linux-arm64
TARBALL=rustbgpd-${SUFFIX}.tar.gz

curl -fL -o "$TARBALL" \
  "https://github.com/lance0/rustbgpd/releases/latest/download/${TARBALL}"
curl -fLO \
  "https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$TARBALL" '$2 == file || $2 == "./" file { print }' \
  "checksums-${SUFFIX}.txt" | sha256sum -c -
tar -xzf "$TARBALL"
sudo install -m 0755 rustbgpd rbgp rs-config-render birdwatcher-adapter /usr/local/bin/
sudo install -m 0644 share/man/man1/rbgp.1 /usr/local/share/man/man1/
sudo install -m 0644 share/man/man8/rustbgpd.8 /usr/local/share/man/man8/
sudo install -m 0644 share/completions/rbgp.bash \
  /usr/share/bash-completion/completions/rbgp

sudo useradd --system --home-dir /var/lib/rustbgpd \
  --shell /usr/sbin/nologin rustbgpd 2>/dev/null || true
sudo install -m 0644 share/systemd/rustbgpd.service \
  share/systemd/rustbgpd@.service /etc/systemd/system/
sudo install -d -m 0755 /etc/rustbgpd
rustbgpd --init-config edge | \
  sudo tee /etc/rustbgpd/config.toml >/dev/null
sudo chmod 0640 /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml

# Kernel-dataplane hosts only; control-plane-only hosts omit this drop-in.
sudo install -d /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 share/systemd/rustbgpd-dataplane.conf \
  /etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.conf

The man pages and completions are also generated on demand by the binaries themselves (rbgp man, rustbgpd --man, rbgp completions bash|zsh|fish), so an installed binary can always regenerate them. Bash and Zsh complete flags at every command level. Fish completes flags only for top-level commands and their direct subcommands; for deeper commands such as rbgp policy chain set-import, it completes the subcommand name but not its flags. All three shells offer --json-lines, --pager, and --json-version only on command paths where those flags apply.

To pin to a specific tag for reproducibility, swap latest for the version, e.g. releases/download/v0.75.0/${TARBALL}. SHA-256 checksums are published alongside each tarball as checksums-${SUFFIX}.txt.

Verify:

rustbgpd --version
rbgp --version
rs-config-render --version

rs-config-render matters only if you run an IXP route server — it turns arouteserver template-context output into rustbgpd configuration (tools/rs-config-render/). It is in the archive either way, so install it now rather than discovering it is missing from a cron refresh later.

For an IXP Manager candidate, run the per-handle rustbgpd@.service instance (its ExecStart reads /var/lib/rustbgpd/<handle>/activation/current/config.toml), then let the renderer publish and settle an immutable generation under that handle:

sudo -u rustbgpd /usr/local/bin/rs-config-render activate \
  --router-handle rs1-ipv4 \
  --candidate /var/lib/rustbgpd/ixp-manager/candidate \
  --runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
  --state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
  --host-state-dir /var/lib/rustbgpd/ixp-manager-host \
  --check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
  --rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
  --activation-command /usr/bin/sudo \
  --activation-arg=-n --activation-arg /usr/bin/systemctl \
  --activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4

Render and activate as the rustbgpd service identity; it must own the private candidate, the per-handle runtime and activation directories, and the shared host-state directory. The runtime directory basename must equal the handle, the activation directory must be exactly <runtime>/activation, and --rbgp-addr must be that runtime's grpc.sock. Authorize that account in sudoers for only the literal /usr/bin/systemctl reload-or-restart rustbgpd@rs1-ipv4. Pre-create the absolute runtime, activation, and host-state directories as non-symlink mode-0700 directories owned by rustbgpd; do not let another process publish or reload during the call. Add --initial only when both no current generation and no reachable daemon exist. Normalized comparison TOML is limited to 4,194,299 bytes (4 MiB minus five encoded-request bytes).

The activation executable must be synchronous. Exit 7 means the candidate was not applied: the executable could not start, or the daemon rejected its reload without runtime effect. The prior link is then restored without another activation and the prior runtime is verified unchanged. The helper accepts a rejection only when rbgp metrics, read before the executable and again after the prior link is restored, shows the same daemon process recorded exactly one SIGHUP outcome, rejected_no_effect, and no runtime-config settlement is in progress. Otherwise, once the executable starts, a nonzero exit, timeout, or unsettled result leaves current on the candidate and returns exit 5; a rejection that cannot be re-proven after the prior link is restored also returns exit 5, with current on the previous generation. Leave retained state untouched and inspect the current private receipt if present; recovery or receipt durability is unproven, and the receipt may be absent or stale when its final write or directory sync failed. rs-config-render status reads that state without changing it and the rs-config-render recover verbs (keep-current, rollback, release-lock, clear; dry run unless --apply) resolve it; the operator procedure for exit 5 is the activation manual-recovery runbook.

To drive the same path directly from IXP Manager v7.4, create a separate mode-0700 candidate directory for the run and store the API key in an absolute mode-0600 regular file owned by rustbgpd:

sudo -u rustbgpd /usr/local/bin/rs-config-render ixp-manager-lifecycle run \
  --ixp-origin https://ixp.example.net --router-handle rs1-ipv4 \
  --api-key-file /var/lib/rustbgpd/ixp-manager/api-key \
  --candidate-dir /var/lib/rustbgpd/ixp-manager/candidate-1 \
  --runtime-state-dir /var/lib/rustbgpd/rs1-ipv4 \
  --state-dir /var/lib/rustbgpd/rs1-ipv4/activation \
  --host-state-dir /var/lib/rustbgpd/ixp-manager-host \
  --max-prefix-restart-seconds 300 \
  --check-with /usr/local/bin/rustbgpd --rbgp /usr/local/bin/rbgp \
  --rbgp-addr unix:///var/lib/rustbgpd/rs1-ipv4/grpc.sock \
  --activation-command /usr/bin/sudo \
  --activation-arg=-n --activation-arg /usr/bin/systemctl \
  --activation-arg reload-or-restart --activation-arg rustbgpd@rs1-ipv4

The helper uses the exact v7.4 lock, Foil configuration, updated, and release endpoints. HTTPS platform roots are required; redirects, proxies, URL credentials, and path-prefixed origins are disabled. Requests send the API key only in the credential header. A synced private journal precedes each remote effect. Exit 6 means one callback is pending and may be retried with ixp-manager-lifecycle resume; exit 5 means acquisition or activation remains uncertain and no callback is attempted automatically — see the activation manual-recovery runbook. Delivery is at-least-once.

Bound retained activation generations

Every distinct activation leaves an immutable generation under <state-dir>/generations/; activation and lifecycle commands never remove one. Run rs-config-render prune --keep N separately. It reports a dry-run plan unless --apply is present.

Pruning refuses with exit 2 while a host fence or competing host command exists, or when retained receipt or journal state cannot be read safely. The host lock remains held through removal, so recovery state cannot appear between planning and deletion. The tool README's pruning section is authoritative for the complete command, stable output, and cron-safe pattern.

rs-config-render exit codes

Every rs-config-render subcommand shares one exit-code table, mirrored from the tool README:

Debian / RPM packages

Tagged releases also publish native packages built with nfpm from the same binaries as the tarballs (packaging/nfpm.yaml):

• rustbgpd_X.Y.Z_{amd64,arm64}.deb — Debian 11+, Ubuntu 22.04+
• rustbgpd-X.Y.Z-1.{x86_64,aarch64}.rpm — RHEL/Rocky/Alma 9+

# Debian / Ubuntu
curl -fLO https://github.com/lance0/rustbgpd/releases/latest/download/rustbgpd_X.Y.Z_amd64.deb
sudo apt-get install -y ./rustbgpd_X.Y.Z_amd64.deb

# RHEL / Rocky / Alma
sudo dnf install -y \
  https://github.com/lance0/rustbgpd/releases/download/vX.Y.Z/rustbgpd-X.Y.Z-1.x86_64.rpm

The package installs the four binaries (rustbgpd, rbgp, rs-config-render, and birdwatcher-adapter) to /usr/bin, the hardened systemd singleton and opt-in per-handle template units (see systemd below — ExecStart pointed at /usr/bin), man pages and shell completions, and creates the rustbgpd system user (imperatively at install time, plus a declarative sysusers.d file). A starter config lands at /etc/rustbgpd/config.toml (mode 0640 root:rustbgpd, never overwritten on upgrade) — the minimal profile with its state paths set to /var/lib/rustbgpd. Version-matched monitoring payloads land under /usr/share/doc/rustbgpd/monitoring/. Then:

sudo $EDITOR /etc/rustbgpd/config.toml   # set ASN, router_id, neighbors
sudo systemctl enable --now rustbgpd

The kernel-dataplane capability drop-in ships as documentation at /usr/share/doc/rustbgpd/examples/rustbgpd-dataplane.conf; copy it into /etc/systemd/system/rustbgpd.service.d/ only for deployments that program the kernel (see below). Config-mutating RPCs (rbgp neighbor add, gNMI Set) need /etc/rustbgpd writable by the service user; the packaged default (0750 root:rustbgpd, read-only for the daemon) supports the SIGHUP-reload workflow — chown rustbgpd /etc/rustbgpd /etc/rustbgpd/config.toml if you want runtime mutation.

Verifying release artifacts

Each architecture has a checksums-<arch>.txt manifest covering its tarball and native packages. Select the exact downloaded filename before passing the row to sha256sum; checking the whole manifest fails when the other package formats have not also been downloaded:

SUFFIX=linux-amd64
ARTIFACT=rustbgpd-linux-amd64.tar.gz
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/${ARTIFACT}"
curl -fLO "https://github.com/lance0/rustbgpd/releases/latest/download/checksums-${SUFFIX}.txt"
awk -v file="$ARTIFACT" '$2 == file || $2 == "./" file { print }' \
  "checksums-${SUFFIX}.txt" | sha256sum -c -

This detects a download that differs from the manifest published with the GitHub release; it does not independently authenticate GitHub. The committed Cargo.lock is the exact Rust dependency inventory for a tagged source tree. Release binaries and containers are provided under the repository's license terms and warranty disclaimers.

Monitoring payloads

Every release archive carries its version-matched Grafana dashboards, Prometheus alert rules, and promtool test suite under share/monitoring/. Native packages put the same files under /usr/share/doc/rustbgpd/monitoring/. Keep rustbgpd-alerts.yml beside rustbgpd-alerts_test.yml: the suite deliberately uses the relative rule_files entry rustbgpd-alerts.yml.

Validate the shipped rules before loading them:

# From an extracted release tarball:
(cd share/monitoring && promtool test rules rustbgpd-alerts_test.yml)

# Or from a native package:
(cd /usr/share/doc/rustbgpd/monitoring && \
  promtool test rules rustbgpd-alerts_test.yml)

Copy or reference rustbgpd-alerts.yml from Prometheus's rule_files configuration, then import rustbgpd-overview.json and, when operating EVPN, the Alpha rustbgpd-evpn.json dashboard in Grafana. The Grafana guide supplies the matching scrape job and dashboard setup; keep the payloads from one rustbgpd release together so rules and metrics do not drift across versions.

From source

# Prerequisites: Rust ≥ 1.95, protobuf-compiler
sudo apt-get install -y protobuf-compiler   # Debian/Ubuntu
git clone https://github.com/lance0/rustbgpd
cd rustbgpd
# Builds exactly what a deployment ships: daemon, rbgp, the route-server
# config renderer, and Birdwatcher adapter. (`--workspace` would also build
# dev/bench helpers you don't need.)
cargo build --release -p rustbgpd -p rustbgpctl -p rs-config-render -p birdwatcher-adapter
sudo install -m 0755 \
  target/release/{rustbgpd,rbgp,rs-config-render,birdwatcher-adapter} \
  /usr/local/bin/

Container image

The current release workflow can build and runtime-verify images natively on Linux amd64 and arm64, then publish a single multi-arch (amd64+arm64) manifest on a tagged release. This does not backfill existing images: :0.68.0 and :0.68 remain amd64-only. :latest follows the newest non-prerelease release and inherits that release's platform set. The workflow configuration produces three tag flavors with docker/metadata-action in .github/workflows/container.yml:

Major-minor is the usual operator default — auto-receives bug-fix releases but pins against minor-version churn. The examples in this document use :latest so the tag stays copy-pasteable across releases. Substitute the major-minor tag of the series you standardize on:

docker pull ghcr.io/lance0/rustbgpd:latest

The published image is the Dockerfile's default runtime target: a lean daemon with rbgp and birdwatcher-adapter, running as a nonroot rustbgpd user. No dev/test/bench helpers are included. rs-config-render is deliberately left out too: the IXP refresh loop is a host-side cron job that runs arouteserver and the renderer next to each other and hands the daemon a finished config directory, so the renderer belongs on the host, not in the daemon's image. Install it from the tarball. If you'd rather build the image locally:

docker build -t rustbgpd:latest .

The development image — adds evpn-tester / evpn-monitor, iproute2, and the interop start script, and runs as root for lab plumbing — is the dev target. It's what the M-series interop suite and the soak harnesses run against:

docker build --target dev -t rustbgpd:dev .

Birdwatcher adapter

Native packages install birdwatcher-adapter at /usr/bin/birdwatcher-adapter; the production container installs it at /usr/local/bin/birdwatcher-adapter. For a same-host deployment, point it at the daemon's absolute Unix socket and run it under a supervisor as the rustbgpd user:

sudo -u rustbgpd birdwatcher-adapter \
  --grpc-addr unix:///var/lib/rustbgpd/grpc.sock \
  --listen 127.0.0.1:8080

TCP remains supported with --grpc-addr http://127.0.0.1:50051. The adapter may start before the daemon/socket: REST requests return 502 Bad Gateway while gRPC is unavailable and recover without restart. No service unit is installed; use the supervisor and lifecycle policy appropriate for the Alice-LG deployment. See the adapter README for bearer-token and endpoint details. This convenient owner-only socket authenticates as the operator-tier local-operator; use a dedicated token-authenticated listener mapped to observer when least privilege is required.

systemd

A hardened unit lives at examples/systemd/rustbgpd.service. It runs the daemon unprivileged by default: a dedicated rustbgpd system user, the standard sandbox set (NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, StateDirectory / RuntimeDirectory), and a capability set of exactly CAP_NET_BIND_SERVICE (bind port 179 as non-root). That is everything a route-reflector / control-plane-only deployment needs.

Releases also ship the opt-in rustbgpd@.service for IXP Manager. Literal %i is the exact router handle: it selects /var/lib/rustbgpd/%i/activation/current/config.toml, a private per-handle runtime/UDS, and no writable access to the shared lifecycle fence. Pre-create each exact handle and the one shared fence directory before the first lifecycle run; authorize and invoke only literal instances, never a wildcard sudoers rule:

handle=rs1-ipv4
sudo install -d -m 0700 -o rustbgpd -g rustbgpd \
  "/var/lib/rustbgpd/$handle" "/var/lib/rustbgpd/$handle/activation" \
  /var/lib/rustbgpd/ixp-manager-host
sudo systemctl enable "rustbgpd@$handle"

Use distinct runtime/activation/UDS paths for every handle and the same /var/lib/rustbgpd/ixp-manager-host for all lifecycle commands on that host.

Notes on the sandbox:

• CAP_NET_BIND_SERVICE lets the daemon bind port 179 without running as root. Nothing else is granted by default — in particular not CAP_NET_ADMIN, so the unprivileged unit cannot program the kernel.

• ProtectSystem=strict + explicit ReadWritePaths confines the daemon to /var/lib/rustbgpd (state) and /etc/rustbgpd (config).

• ExecReload=kill -HUP is the supported reload path. See the reload matrix for which fields hot-apply vs. need a restart.

• Type=notify is load-bearing. The daemon sends READY=1 only after every configured gRPC listener is bound, the configured peer roster is installed, and BGP ingress is active — the same boundary /readyz cannot go green before — so systemctl start and units ordered After=rustbgpd.service wait for a daemon that can accept sessions, not for a forked process. A gRPC bind failure prevents peer registration, BGP ingress, telemetry, dial-out, and readiness publication before entering shortened coordinated teardown. STOPPING=1 marks the start of coordinated shutdown. With WatchdogSec=5min the daemon sends WATCHDOG=1 every 2.5 minutes while the PeerManager and RIB actors answer the same bounded core-actor probe /readyz uses. A missed probe skips the ping and retries every second; PID 1 independently expires the service after five minutes without a ping, and Restart=on-failure covers that expiry. The probe's 200 ms timeout bounds one liveness sample; it is not a separate service deadline. TimeoutStartSec=10min bounds initial roster installation before READY=1.

  Without NOTIFY_SOCKET — a foreground run or Docker — every notification is a silent no-op. A native unit opting out must disable all inherited notify behavior:

  [Service]
  Type=simple
  NotifyAccess=none
  WatchdogSec=0

  The container unit keeps Type=simple because Docker, not the daemon, is its main process.

• Restart=on-failure is load-bearing. The daemon exits 0 only on an operator-initiated shutdown (SIGINT/SIGTERM, the Shutdown RPC) and 1 on a component failure it cannot recover from in place. Startup exits immediately if legacy BGP mode binds neither family, explicit listen_addresses cannot bind every configured endpoint, or a configured prometheus_addr health listener cannot bind, or any configured gRPC listener cannot bind. An unexpected gRPC server, RIB manager, peer manager, RPKI subsystem task, BGP listener task, inbound accept-forwarding task, or metrics/readiness server exit instead runs the coordinated peer teardown before exit 1. These components are not respawned in place; the supervisor is the recovery path. Exit 70 is also a failure: it is the runtime-config settlement watchdog's fail-stop recovery request and must remain restartable. RestartSec=5 retries a transient failure, while StartLimitBurst=5 and StartLimitIntervalSec=10min stop a deterministic failure from flapping forever. After correcting config authority or permissions, recover a rate-limited unit with systemctl reset-failed rustbgpd followed by systemctl start rustbgpd. An explicit systemctl stop suppresses restart, and TimeoutStopSec=32min gives an already-owned mutation longer than its 30-minute watchdog plus five-second terminal grace to settle or fail-stop. Nothing else in the stop path needs that long: waits outside settlement ownership are bounded in seconds, and a second SIGTERM or SIGINT skips the ones that have no deadline, so TimeoutStopSec does not need to change. A second signal while a mutation is still settling instead forces the watchdog's fail-stop at once (fence_reason="operator_forced", exit 70 after the grace). The supervisor consequence depends on how the stop was requested: raw signals to a running unit end in exit 70, an unclean exit code that Restart=on-failure restarts, while a unit stopped explicitly with systemctl stop is never restarted automatically whatever its exit status. Recovery from the persisted transaction runs on the next actual start either way. rbgp doctor reports listener failures through its bgp.listener check.

Installation

The Debian/RPM packages perform all of the below on install (unit at /lib/systemd/system, ExecStart at /usr/bin). The tarball commands above use its share/systemd/ tree. The following paths are only for a source checkout:

sudo useradd --system --home-dir /var/lib/rustbgpd \
  --shell /usr/sbin/nologin rustbgpd
sudo install -m 0644 examples/systemd/rustbgpd.service \
  /etc/systemd/system/rustbgpd.service
sudo install -d -m 0755 /etc/rustbgpd
sudo install -m 0640 examples/minimal/config.toml /etc/rustbgpd/config.toml
sudo chgrp rustbgpd /etc/rustbgpd/config.toml   # readable by the service user
$EDITOR /etc/rustbgpd/config.toml    # set ASN, router_id, neighbors
sudo systemctl daemon-reload
sudo systemctl enable --now rustbgpd

Kernel dataplane: opt-in privilege drop-in

If — and only if — the config programs the kernel ([[fib_tables]] per ADR-0061, install_blackhole_discard, or the EVPN VTEP/IRB dataplane), the daemon needs CAP_NET_ADMIN for rtnetlink route / nexthop programming, VXLAN + bridge FDB writes, and the RTNLGRP_NEIGH subscription behind local-MAC learning. Grant it via the shipped drop-in examples/systemd/rustbgpd-dataplane.conf instead of editing the base unit — systemd merges capability sets, so the drop-in adds CAP_NET_ADMIN on top of the unprivileged default:

sudo mkdir -p /etc/systemd/system/rustbgpd.service.d
sudo install -m 0644 examples/systemd/rustbgpd-dataplane.conf \
  /etc/systemd/system/rustbgpd.service.d/rustbgpd-dataplane.conf
sudo systemctl daemon-reload
sudo systemctl restart rustbgpd

RR-only boxes should not install this drop-in — the unprivileged default is the point.

Dataplane host caveat: if rustbgpd programs the kernel on a host that also runs systemd-networkd, set ManageForeignRoutes=no, ManageForeignRoutingPolicyRules=no, and (where supported) ManageForeignNextHops=no in networkd.conf. The defaults are yes, and networkd will delete routes and next-hop groups it did not create — including rustbgpd's, especially across a daemon restart (ADR-0079). Co-residency with another daemon that claims proto bgp kernel state (e.g. FRR zebra) is unsupported for the same reason.

Reserved nexthop-ID ranges (EVPN dataplane)

When the EVPN VTEP/IRB dataplane is enabled, rustbgpd exclusively owns four kernel nexthop-ID ranges, distinguished by the ID's high nibble (deliberately offset from FRR's 0x1000/0x2000 tags so both daemons can't collide on NLM_F_REPLACE):

This is a deployment contract: no co-resident netlink writer may create nexthop objects with IDs in these ranges (ip nexthop add id …, another routing daemon, orchestration tooling). rustbgpd adopts objects it finds there across restarts and garbage-collects the ones nothing references — an unrelated agent's object parked in the range would be treated as rustbgpd state.

rustbgpd validates the shape of every in-range object it dumps at startup adoption and during drift recovery (FDB flag, member vs group structure matching the tag). An in-range object that does not look like anything rustbgpd writes fails closed per ID: the object is left untouched, excluded from adoption and from every reap/delete path, its ID is quarantined so the allocator can never hand it out (and thus never NLM_F_REPLACE over it), and the violation is logged once and counted on the evpn_foreign_nhid_range_conflicts_total Prometheus counter. A nonzero counter means a co-resident writer is violating this contract — find and remove that writer; the quarantined IDs are not reclaimed until a daemon restart after the foreign objects are gone.

Note this shape check cannot distinguish a foreign object that is byte-for-byte identical to what rustbgpd writes; stamping nh_protocol on owned nexthop objects as a provenance marker is a possible future defense-in-depth, not a substitute for this contract (any netlink writer can set the same protocol value — it is provenance, not kernel-enforced exclusivity).

Operational checks

systemctl status rustbgpd            # is it running?
journalctl -u rustbgpd -f            # follow logs
systemctl reload rustbgpd            # SIGHUP → config reload
systemctl restart rustbgpd           # full restart (state drained)

Docker / Docker Compose

A worked starting point at examples/docker-compose/ brings up rustbgpd alongside an FRR peer that advertises sample prefixes — no real routers required.

cd examples/docker-compose
docker compose up -d --build
docker compose exec rustbgpd \
  rbgp -s http://127.0.0.1:50051 neighbor
docker compose down

For your own deployment:

• Image name and command: published GHCR version tags have no leading v: use ghcr.io/lance0/rustbgpd:0.75.0, not ghcr.io/lance0/rustbgpd:v0.75.0. The older :0.68.0 image is amd64-only; it is not backfilled when a later release publishes multiple platforms. The production image runs as uid/gid 999 — pinned in the Dockerfile runtime stage (useradd --uid 999 --gid 999 and USER 999:999), not allocated by the base image, and asserted against the built image by the container workflow — and its default command is exactly rustbgpd /etc/rustbgpd/config.toml. If the config is mounted under another container filename, pass that filename explicitly after the image, for example rustbgpd /etc/rustbgpd/router.toml.

• Bridge networking and state: mount a volume at the daemon's runtime_state_dir so the GR restart marker and FIB and BLACKHOLE ownership receipts survive container restarts. The daemon runs as uid/gid 999 and rewrites its config in place, so both host directories must be owned by that uid and the config must already exist — the image's default command is rustbgpd /etc/rustbgpd/config.toml and it will not create one:

  # One-time host prep.
  sudo install -d -o 999 -g 999 /etc/rustbgpd /var/lib/rustbgpd
  docker run --rm ghcr.io/lance0/rustbgpd:latest \
    rustbgpd --init-config edge > config.toml
  # Edit config.toml for your ASN, router ID, peers, and policy.
  sudo install -o 999 -g 999 -m 0600 config.toml /etc/rustbgpd/config.toml
  
  docker run --rm -d \
    --name rustbgpd \
    --network=bridge \
    --stop-timeout=1920 \
    -v /etc/rustbgpd:/etc/rustbgpd \
    -v /var/lib/rustbgpd:/var/lib/rustbgpd \
    -p 179:179 \
    -p 9179:9179 \
    --ulimit nofile=65536:524288 \
    ghcr.io/lance0/rustbgpd:latest

  This recipe succeeds because Docker sets net.ipv4.ip_unprivileged_port_start=0 inside its default bridge network namespace. Port 179 is therefore not privileged there; no capability flag is responsible for the bind. Add NET_ADMIN only when the container actually programs its network namespace. The bridge path is the supported choice when the daemon does not need to bind a particular host/peering-LAN address.

  The edge profile is used because its runtime_state_dir, grpc_uds path, and 0.0.0.0 metrics bind are already the container forms; the lab profile's /tmp state directory and 127.0.0.1 metrics bind both need editing first.

Host networking and privileged BGP ports

Host networking is the second supported container path, and is required when listen_addresses must name a host/peering-LAN address, a multihop session pins its source, or the daemon must program the host network namespace. It does not inherit Docker's bridge-netns low-port setting:

--cap-add=NET_BIND_SERVICE is not sufficient for the image's uid 999 in host mode. Docker does not provide that capability as an ambient capability to the non-root image process, so it is unavailable after the uid 999 exec. This differs from the bare-binary systemd unit above, whose AmbientCapabilities=CAP_NET_BIND_SERVICE is effective and remains correct.

Use exactly one of these deployment paths:

1. Keep the default bridge network and published ports when address pinning is unnecessary; the image continues to run as uid 999.

2. Use --network=host with --user=root and bind a root-owned host state directory at /var/lib/rustbgpd:

   sudo install -d -o root -g root -m 0755 /etc/rustbgpd
   sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd
   sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml
   
   docker run --rm -d \
     --name rustbgpd \
     --network=host \
     --user=root \
     --cap-drop=ALL \
     --cap-add=NET_BIND_SERVICE \
     --stop-timeout=1920 \
     --ulimit nofile=65536:524288 \
     --mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \
     --mount=type=bind,source=/var/lib/rustbgpd,target=/var/lib/rustbgpd \
     ghcr.io/lance0/rustbgpd:0.75.0

Do not omit the host state bind when overriding the image user. The directory baked into the image is owned by uid 999; a root daemon expects uid 0. The runtime-state owner guard treats that mismatch as unsafe, logs runtime-state marker/checkpoint storage unavailable, and disables the graceful-restart marker and warm checkpoint storage. v0.69.0 and later images also reject the default gRPC UDS listener at startup. A root-owned, non-group/world-writable bind mount makes the authority match the running uid. The socket parent must allow the daemon read/write/search access, with no symlinks or untrusted writable ancestors; see Unix socket path integrity.

Adding NET_BIND_SERVICE to uid 999 is not a third supported host-network recipe. If changing the host's low-port sysctl is part of a separately managed host policy, validate that policy outside this unit; the shipped container guidance does not mutate the host sysctl.

The root host-network path drops Docker's entire default capability set, then adds back only NET_BIND_SERVICE for port 179. Root makes that explicitly granted capability usable; applying the same add-cap flag to uid 999 remains ineffective because Docker supplies no ambient capability to that non-root exec.

The command above and the systemd unit below are control-plane defaults. A configuration that programs the host FIB or interfaces also needs --cap-add=NET_ADMIN on the same root host-network path, as described in kernel dataplane privileges; that opt-in does not replace the root-user and root-owned-state requirements.

systemd-supervised host-network container

examples/systemd/rustbgpd-container.service generalizes the host-network recipe above. It keeps docker run attached so the daemon's exit status drives Restart=on-failure, maps systemctl reload to SIGHUP, gives Docker the full 32-minute settlement grace and systemd a 33-minute outer margin, and makes the registry pull best-effort so an explicitly selected cached image remains usable during a registry outage. A missing or empty RUSTBGPD_IMAGE fails closed before any Docker command runs.

Install the unit and its required image environment file after creating the root-owned state directory:

sudo install -d -o root -g root -m 0755 /etc/rustbgpd
sudo install -d -o root -g root -m 0700 /var/lib/rustbgpd
sudo install -o root -g root -m 0600 config.toml /etc/rustbgpd/config.toml
sudo install -m 0644 examples/systemd/rustbgpd-container.service \
  /etc/systemd/system/rustbgpd-container.service
sudo tee /etc/rustbgpd/rustbgpd-container.env >/dev/null <<'EOF'
RUSTBGPD_IMAGE=ghcr.io/lance0/rustbgpd:0.75.0
EOF
sudo chmod 0644 /etc/rustbgpd/rustbgpd-container.env
sudo systemctl daemon-reload

The overridable host defaults are RUSTBGPD_CONFIG_DIR=/etc/rustbgpd and RUSTBGPD_STATE_DIR=/var/lib/rustbgpd; both are bind-mounted at their standard container paths. RUSTBGPD_CONFIG_FILE=/etc/rustbgpd/config.toml selects the file inside the read-only config mount. Put any overrides beside RUSTBGPD_IMAGE in the environment file. An upgrade is one image-tag edit followed by systemctl restart rustbgpd-container; version tags still omit the leading v. RUSTBGPD_IMAGE is required and must be non-empty; the unit checks it before its best-effort pull.

The unit deliberately mounts the config directory read-only. Runtime mutation (rbgp neighbor add, policy edits, gNMI Set, and rbgp config apply) is therefore excluded: those calls cannot persist and are rejected. Supporting them requires an explicit change to the mount and root-owned config-directory write contract; the shipped unit is for externally managed config plus SIGHUP.

Before the first start, and before restarting after an image or config change, validate the exact externally managed config directory and selected image without touching the running container:

docker run --rm \
  --user=root \
  --cap-drop=ALL \
  --mount=type=bind,source=/etc/rustbgpd,target=/etc/rustbgpd,readonly \
  ghcr.io/lance0/rustbgpd:0.75.0 \
  rustbgpd --check --strict /etc/rustbgpd/config.toml

Only enable the service after that check succeeds:

sudo systemctl enable --now rustbgpd-container

The image's declared healthcheck remains active under systemd supervision and runs rbgp health --liveness (gRPC handler liveness, not core-actor readiness) against the default state-directory UDS. An unhealthy result remains observable with docker inspect --format '{{.State.Health.Status}}' rustbgpd, but Docker health status does not terminate the attached process and therefore does not drive systemd's Restart=on-failure. If the config moves the socket, set RUSTBGPD_ADDR for the container by extending the unit's docker run command.

• File descriptors: --ulimit nofile sets the ceiling. rustbgpd raises its soft limit to the hard limit at startup, so the hard limit must clear the 4096 floor rbgp doctor enforces; peers exhaust descriptors at scale.

• Writable config directory: mount /etc/rustbgpd read-write and make it owned by the daemon user (chown the host directory to the container's rustbgpd uid) if you use runtime mutation — rbgp neighbor add, policy edits, gNMI Set, rbgp config apply. Config persistence rewrites the file with a temp-file + rename, so the directory, not just the file, has to be writable, and every mutating RPC is rejected without it. Mount it :ro only when the config is managed entirely from outside and reloaded with SIGHUP.

• Settlement fail-stop: --stop-timeout=1920 gives the daemon a 32-minute Docker stop grace; the shipped container unit's TimeoutStopSec=33min is the supervisor margin around that operation. A failure to create the initial persistence stage rejects cleanly before runtime mutation. After runtime mutation begins, a failed rename is NotPublished: only complete acknowledged compensation returns cleanly; otherwise known divergence fences recovery. A post-rename directory-fsync failure is PublicationAmbiguous: the complete visible candidate is adopted without rollback, and restart selects durable authority. A read-only bind-mounted config file can have a writable parent yet still fail replacement as NotPublished. A fenced daemon marks readiness unavailable, closes persisted-mutation admission, and exits 70 within five seconds; a silent owner is fenced at 30 minutes and exits five seconds later. This standalone command has no restart policy; use bounded supervisor retries and inspect durable config authority before recovery.

• Health: the image runs rbgp health --liveness every 30 seconds. Success means only that the authenticated gRPC handler answered; it does not check core-actor readiness, BGP convergence, or forwarding. This is an intentional change from the previous image probe, which used GetHealth and checked core actors. To retain that behavior, start the container with --health-cmd='rbgp health'. Ordinary rbgp health keeps its detailed output and readiness behavior. The liveness RPC requires only the read listener tier; credentials and principal-role checks still apply. Failed, refused, or unreachable probes exit non-zero. The default endpoint is unix:///var/lib/rustbgpd/grpc.sock; set RUSTBGPD_ADDR on the container if configuration moves it. HTTP /livez is also available on the metrics listener; /readyz retains core-actor readiness checks.

• Logs: [global.telemetry] log_format = "json" emits structured JSON. Bound retention when creating the container, for example with --log-driver=json-file --log-opt max-size=10m --log-opt max-file=3 on docker run. Add these flags to the container systemd unit's docker run command too. For Compose, set logging.driver: json-file and logging.options: {max-size: "10m", max-file: "3"} on the service. These limits keep approximately 30 MB per container; recreate existing containers to apply them. The image cannot configure the host logging driver. Forward logs to an aggregator when longer retention is needed. Successful read authorization audits are DEBUG; counters and existing denied, error, sensitive-read, and mutation audit levels are unchanged.

• Networking: Linux FIB integration and BFD require access to the network namespace they operate on. For host addresses, use the root host-network path above; if the configuration also programs the host FIB or interfaces, add its documented NET_ADMIN opt-in. A bridge container affects only its own namespace. The interop suite runs rustbgpd as a containerlab kind: linux node, which is the cleanest reference setup for an isolated lab namespace.

Containerlab quick start

Containerlab is the easiest way to get a working rustbgpd ↔ FRR session on your laptop with no real routers. The simplest topology in the repo is tests/interop/m0-frr.clab.yml:

name: m0-frr
topology:
  nodes:
    rustbgpd:
      kind: linux
      image: rustbgpd:dev
      cmd: sleep infinity
    frr:
      kind: linux
      image: quay.io/frrouting/frr:10.7.1
  links:
    - endpoints: ["rustbgpd:eth1", "frr:eth1"]

Bring it up:

# Build the dev image
docker build --target dev -t rustbgpd:dev .

# Deploy
sudo containerlab deploy -t tests/interop/m0-frr.clab.yml

# Inspect (the test driver scripts under tests/interop/scripts/ show
# the typical incantations for FRR vtysh + rbgp gRPC).

# Tear down
sudo containerlab destroy -t tests/interop/m0-frr.clab.yml --cleanup

For more topologies (route-reflector, EVPN, FlowSpec, BFD, etc.), see INTEROP.md.

Recommended first production-ish topology

The smallest deployment that exercises the full operator loop — build → validate → reload → observe — without depending on more than one peer. This is your "did I install this correctly" gate before scaling out.

1. One rustbgpd instance + one FRR peer (or any compliant BGP peer). Provision the peer first; record its address + AS.

2. Prometheus scrape on :9179. Add the scrape target in your Prometheus config:

   - job_name: rustbgpd
     static_configs:
       - targets: ["10.0.0.1:9179"]

3. Config validation. Build the config, then:

   rustbgpd --check /etc/rustbgpd/config.toml

   Errors are rustc-style with file + line + carets; expect config OK on success. Run this every time before swapping the live config.

   A check that finds nothing to flag prints config OK. A check that flags something still exits 0, but the summary reads config VALID, <n> WARNINGS — NOT a clean check and the warnings are framed on stderr above it. The warnings identify: a configured eBGP neighbor or dynamic range with no explicit policy in a direction (unfiltered when [global] ebgp_requires_policy is off, carrying no routes in that direction when it is on); an RFC 8212 posture still inherited from legacy omission (rfc8212_secure_default_ready — set an explicit root config_epoch plus [global] ebgp_requires_policy); and an orr_vantage with no linkstate-negotiating neighbor. None is rejected — a permit-all route server is a legitimate configuration — but none should reach production unnoticed.

4. First start.

   sudo systemctl start rustbgpd
   journalctl -u rustbgpd -f

   Watch for the session to reach Established:

   rbgp neighbor

5. Edit + reload cycle. Edit a copy of the config, then dry-run the diff of the candidate against the current on-disk config (--diff takes the candidate; the positional argument is the config to compare against, default /etc/rustbgpd/config.toml):

   cp /etc/rustbgpd/config.toml /tmp/new-config.toml
   $EDITOR /tmp/new-config.toml
   rustbgpd --diff /tmp/new-config.toml /etc/rustbgpd/config.toml

   When the diff looks right, install the candidate over /etc/rustbgpd/config.toml before reloading.

   --diff calls into the same ConfigDiff machinery the reload path uses; what it reports is what reload will do. Cross-reference against the reload matrix to confirm which changes hot-apply and which are restart-required. Apply:

   sudo systemctl reload rustbgpd

6. Restart cycle. Confirm the FRR peer reconverges within the GR timer (GR is on by default):

   sudo systemctl restart rustbgpd
   # FRR should retain the routes during the restart window; rustbgpd
   # advertises R=1 in the GR capability on the next OPEN.

7. Observability sanity-check. Read at least one Prometheus counter and one neighbor field:

   curl -s http://10.0.0.1:9179/metrics \
     | grep -E "bgp_session_established_total|bgp_messages_received_total"
   rbgp neighbor 10.0.0.2

If all six steps work end-to-end, your install is sound. Scale from there: add more peers, add policy chains, wire BMP / gNMI / MRT according to your needs.

Config validation workflow

These cover the config lifecycle, from bootstrap to reload:

The validation pipeline is the same in all three places: TOML parse → validate() → ConfigDiff. A config that passes --check will not error on --diff; a clean --diff will not error on reload.

Observability

Prometheus

The exporter binds at [global.telemetry] prometheus_addr. The same listener serves /livez and /readyz for orchestrators. Key counters operators watch:

• Session liveness — bgp_session_established_total, bgp_session_flaps_total, bgp_messages_received_total, bgp_messages_sent_total (all by peer).
• Loop / leak detection — bgp_as_path_loop_detected_total, bgp_rr_loop_detected_total, bgp_otc_routes_blocked_total{peer, reason} (RFC 9234 / ADR-0071), bgp_role_mismatch_total{peer, local_role, remote_role}. Canonical reason values are documented in docs/reference/operations.md ("Ingress rejection / route-leak detection").
• Route processing — bgp_rib_prefixes{peer, afi_safi}, bgp_rib_loc_prefixes{afi_safi}, bgp_max_prefix_exceeded_total.
• GR / FIB — bgp_gr_active_peers, bgp_gr_stale_routes, bgp_fib_routes_installed_total, bgp_fib_kernel_failures_total.
• Allocator — jemalloc_allocated_bytes, jemalloc_active_bytes, jemalloc_resident_bytes, jemalloc_mapped_bytes. Present in builds with the jemalloc feature (the published container image and release tarballs); refreshed at scrape time. allocated is live application bytes, resident is jemalloc's contribution to RSS — a widening gap between the two is retained-but-unused allocator memory, the first thing to check before suspecting a leak. This jemalloc build takes run-time options from _RJEM_MALLOC_CONF, not MALLOC_CONF; see heap profiling with jemalloc.
• Durable event outbox (ADR-0072) — bgp_event_outbox_committed_total{category}, bgp_event_outbox_dropped_total{category, reason}, bgp_event_outbox_db_size_bytes, bgp_event_outbox_latest_event_id, bgp_event_outbox_degraded, bgp_event_outbox_storage_failed, bgp_event_outbox_cursor_gap_total. The degraded gauge is the alert-on-this signal — flips to 1 on a durability-impacting drop, decode/codec failure, or open failure since process start; does not auto-clear in v1, so any non-zero value warrants investigation. The cursor-gap counter alerts on collectors whose persisted last_seen_event_id fell below the daemon's retention floor — typically a sign the collector was offline longer than [event_history].max_events / max_bytes planned for. External collectors stream the cursor via the SubscribeFromEvent gRPC RPC; examples/event-bridge/ is the reference skeleton — copy and replace the stdout writer with your Kafka / NATS / Vector / journald sink, persisting last_seen_event_id after the sink confirms durable receipt. See [event_history] in CONFIGURATION.md for tuning and recovery semantics, and the "Durable Event Cursor" section in OPERATIONS.md for the alert + sizing playbook.

Policy filtering visibility — Prometheus. bgp_policy_routes_total {peer, policy, direction, action} attributes each import and export policy evaluation to the decisive member on Deny. A nonempty chain that completes without rejection uses the bounded policy="chain_default_permit" sentinel; policy="inline" remains the label for an inline Deny or a Permit from an absent / genuinely empty chain. Initial table dumps, route refreshes, dirty resyncs, and forced outbound refreshes can increment export counters because they re-evaluate export policy. Each label identity remains monotonic while its policy/action is reachable from the installed chain. After a successful policy replacement, an unreachable identity is retired and goes stale; reinstalling that exact identity creates a fresh counter starting from zero. Prometheus rate() and increase() handle the resulting counter reset.

# Routes denied by a named filter on each peer's import side:
rate(bgp_policy_routes_total{direction="import", action="deny"}[5m])

An operator policy literally named chain_default_permit remains ordinary Deny attribution. The sentinel interpretation applies only when action="permit".

Policy evaluation errors — Prometheus. bgp_policy_eval_errors_total {direction, kind} counts routes denied by the fail-closed evaluation- error rail (ADR-0103 Decision 4: checked-arithmetic failure, absent operand, fuel/loop-cap exhaustion, ...). These denies also appear in bgp_policy_routes_total as action="deny"; this counter separates "the policy said no" from "the policy is broken". Any nonzero rate deserves a look — rbgp policy stats --direction both names the failing chain, policy, and term (eval_errors count + last_error per chain), and the rate-limited daemon WARN carries the same blame line.

# Any policy erroring anywhere is alert-worthy:
sum by (kind) (rate(bgp_policy_eval_errors_total[5m])) > 0

Policy filtering visibility — gRPC scalar aggregates. NeighborState carries four per-peer running totals to give operators a cheap sanity-check on the labeled Prometheus counter:

Both directions reset together on the next session establishment, so "how many routes did this session permit / deny in total" is straight subtraction; "how many across reconnects" requires Prometheus history. The CLI surfaces these in rbgp neighbor <peer> as a Policy Stats block:

Policy Stats:
  Import:                 permitted=1247 denied=31
  Export:                 permitted=892 denied=0

JSON output (rbgp --json neighbor <peer>) carries the same fields under import_policy_routes_permitted / import_policy_routes_denied / export_policy_routes_permitted / export_policy_routes_denied, elided when zero.

BMP

If [bmp] is configured, rustbgpd opens a TCP session to the BMP collector and exports per-peer Adj-RIB-In + Peer Up / Peer Down events. See CONFIGURATION.md for the schema.

gNMI

The gNMI adapter (ADR-0070) exposes Capabilities / Get / Subscribe telemetry plus the static numbered-neighbor Set subset over the same sockets as the native API. Native gNMI is registered on [global.telemetry.grpc_tcp] only when native mTLS is configured; the [global.telemetry.grpc_uds] listener serves gNMI unconditionally as a local-only extension. RFC 7951 JSON encoding. See GNMI.md for the path namespace and supported mutation leaves.

CLI introspection

rbgp is the operator interface for read queries.

rbgp neighbor                  # list all neighbors
rbgp neighbor 10.0.0.2         # detail
rbgp neighbor 10.0.0.2 --compare 10.0.0.3  # live update-group relationship
rbgp rib                       # browse Loc-RIB
rbgp bfd                       # BFD sessions (ADR-0067)
rbgp evpn                      # EVPN instances + Type 2/3 RIB
rbgp top                       # live TUI dashboard

Most data-oriented read commands support --json for scripting. Commands with fixed formats, such as metrics, completions, and top, keep their command-specific output.

For an Established neighbor, rbgp neighbor <addr> reports whether the peer negotiated Route Refresh, Enhanced Route Refresh, and Extended Messages, plus the directional maximum BGP message size rustbgpd may send (4096 or 65535 bytes). The JSON fields preserve false as an explicit negotiated result and omit values an older daemon did not expose; a missing live-session snapshot is reported separately by negotiation_available.

Package upgrade and rollback

This is the canonical procedure for the release .deb and .rpm packages. The package scripts create the service user and run systemctl daemon-reload; they deliberately do not stop, start, or restart rustbgpd. An operator must therefore own the graceful stop and the verification that follows it.

Upgrade

1. Read the target release's CHANGELOG section and verify the downloaded package against its release checksum. Config compatibility is not assumed across minor releases.

2. Extract the package into a temporary directory and run the new binary's strict check against the configuration that the service will use:

   candidate_pkg=/path/to/rustbgpd_X.Y.Z_amd64.deb # or the .rpm
   candidate_root=$(mktemp -d)
   chmod 0755 "$candidate_root"
   
   case "$candidate_pkg" in
     *.deb) dpkg-deb --extract "$candidate_pkg" "$candidate_root" ;;
     *.rpm) rpm2cpio "$candidate_pkg" |
              (cd "$candidate_root" && cpio --quiet -id --no-absolute-filenames) ;;
     *) echo "expected a .deb or .rpm" >&2; exit 2 ;;
   esac
   
   sudo -u rustbgpd \
     "$candidate_root/usr/bin/rustbgpd" --check --strict \
     /etc/rustbgpd/config.toml

   Fix every error before proceeding. There is no general schema-migration tool; make release-note field renames by hand, then repeat the candidate check. The one exception is the RFC 8212 posture representation, which rustbgpd --migrate-config pin-legacy|prepare-secure|downgrade-v0.64 --offline [--dry-run] CONFIG_PATH rewrites in place (see CONFIGURATION.md → config_epoch); stop or quiesce the daemon first — --offline is an operator assertion, not a daemon probe. This check validates candidate config bytes only: it does not inspect runtime_state_dir or config-adjacent commit-confirm authority.

   For the import explain and rejected-route retention capacity limit, older configs with [policy.explain] cache_size or [policy.reject_retention] capacity above 2,097,152 now fail startup and reload validation, even if that section has enabled = false. Reduce each named value to at most 2,097,152, then run the new binary's --check --strict against the edited file before stopping or restarting the daemon. The defaults remain 4096 and 1024; zero still acts as one.

   Before any upgrade, use the still-running daemon to run the pre-upgrade diagnostic against the file the new binary will boot:

   sudo -u rustbgpd "$candidate_root/usr/bin/rbgp" doctor --pre-upgrade /etc/rustbgpd/config.toml

   It is red, with the next action, while a confirmed transaction is pending, applying, rollback-failed, or ambiguous (confirm or abort it with rbgp config confirm <id> / rbgp config abort <id>), while a runtime-config settlement owner is still settling, when the file resolves to a different RFC 8212 epoch/posture than the live daemon runs, and whenever the evidence is unavailable or denied. It never confirms, aborts, rewrites, or stops anything. A green result is an observation at one instant (human output says Pre-upgrade observation as of unix <t>; JSON carries observed_at_unix_seconds), not a fence: a transaction can still start after it, which is why step 3 stops the daemon and then repeats these checks. See the check reference.

   When the installed release is v0.64.0 or earlier, also clear retired commit-confirm authority first: a locator-free commit-confirm-journal.json or a retired v2 *.commit-confirm-locator.json must be recovered with exactly rustbgpd v0.64.0. Do not proceed while either artifact is present or inaccessible; delete one only after proving the transaction terminal and the current config intended. v0.65.0 and every later release otherwise refuse boot and leave the authority untouched.

3. Stop the running daemon cleanly, then confirm it is down:

   sudo systemctl stop rustbgpd
   systemctl is-active rustbgpd # expected: inactive

   The coordinated stop writes the GR restart marker. With GR enabled, the new process can advertise R=1 while sessions rebuild. Families with configured kernel installers (FIB, blackhole discard, EVPN) still advertise forwarding_preserved = false, while control-plane-only families advertise F=1 under the role rules. This is not a forwarding-continuity guarantee; use a drained route-server pair when traffic continuity matters.

   With the daemon inactive, repeat the checks that only a stopped daemon can make authoritative: verify the absence of a config-adjacent *.commit-confirm-locator.json, make any explicit offline posture correction the check called for, then repeat the candidate --check --strict from step 2 against the final file after every rewrite. The live diagnostic observed an earlier instant; these post-stop checks are the ones that hold.

4. Install the already-checked package:

   sudo apt-get install -y "$candidate_pkg" # Debian / Ubuntu
   sudo dnf install -y "$candidate_pkg"     # RHEL / Rocky / Alma

   Use the command for the host's package manager, not both. The package marks /etc/rustbgpd/config.toml as config|noreplace: a locally modified config stays in place. Inspect any replacement candidate reported by the package manager before starting (normally .dpkg-dist or .rpmnew):

   sudo find /etc/rustbgpd -maxdepth 1 -type f \
     \( -name '*.dpkg-dist' -o -name '*.rpmnew' \) -print
   sudo -u rustbgpd /usr/bin/rustbgpd --check --strict \
     /etc/rustbgpd/config.toml

5. Start and verify the installed version and its operator surfaces:

   sudo systemctl start rustbgpd
   sudo systemctl --no-pager --full status rustbgpd
   /usr/bin/rustbgpd --version
   sudo -u rustbgpd rbgp health
   sudo -u rustbgpd rbgp summary

   Confirm the reported version is the package you selected and wait for every expected session to return to Established; systemd active alone does not prove routing convergence.

Rollback

A rollback is another compatibility migration, not merely a package command. Read the target version's release notes. Extract the old package as above and run its rustbgpd --check --strict against the intended rollback config before stopping the current daemon. If the old binary rejects the current config, restore a version-controlled config known to that release, preflight that exact file with the old binary, and install it only after the stop.

Canonical persistence omits genuinely optional sections while they remain at their defaults. This keeps routine runtime changes from adding unused feature tables to the file. Omitted values use the receiving release's defaults, as they do in hand-written TOML; this is not a cross-release semantic round-trip or downgrade compatibility promise. Once a new feature is configured, its section is retained; renamed fields use the current canonical spelling. An older binary can therefore still reject a newer persisted config. Restoring an older version-controlled config also discards API-driven changes made since that saved copy, so reconcile those changes explicitly before the rollback.

Before downgrading, run rbgp doctor --pre-upgrade against the intended rollback config (or rbgp config status) and finish a pending confirmed transaction with rbgp config confirm <id> or rbgp config abort <id>. Do not delete a live locator to force the downgrade. Once the transaction is terminal, verify that the config-adjacent *.commit-confirm-locator.json and the retired locator-free commit-confirm-journal.json are absent. The fixed v3 raw prior and metadata may remain after a warning-only cleanup failure: once locator absence is verified they are non-authoritative, and rustbgpd v0.64.0 ignores those commit-confirm v3-only names. If the target predates either retained history format, preserve and move the complete config-history/ directory aside after stopping the daemon and before installing the older binary. In particular, writable downgrade after a config-history v3 row has been published is unsupported without moving the directory aside: the older writer ignores v3, can expose stale rows, and may reuse its sequence. There is no backfill or down-conversion. Re-upgrade fails closed on duplicate sequences or an over-cap roster; it cannot make an old writer's collisions trustworthy. Builds that recognize newer-format history rows (see ADR-0124) list them as unreadable and skip recording, with a warning, until they are moved aside.

Then stop cleanly, move history aside if required above, and install the selected older package:

previous_pkg=/path/to/rustbgpd_W.X.Y_amd64.deb # or the .rpm
sudo systemctl stop rustbgpd
# Preserve and move config-history/ aside here if the target cannot read it.
sudo apt-get install -y --allow-downgrades "$previous_pkg" # Debian / Ubuntu
sudo dnf install -y "$previous_pkg"                       # RHEL / Rocky / Alma

Inspect any .dpkg-dist/.rpmnew, install the preflighted compatible config, then repeat the start and verification commands from the upgrade procedure. A binary that predates GR marker v3 rejects the v3 marker and cold-starts without restarting-speaker mode. Do not count on peer route retention across that boundary; drain the route-server member of a redundant pair first when continuity matters.

Persistent state on disk

Runtime state lives in runtime_state_dir (default /var/lib/rustbgpd). The v3 commit-confirm locator instead lives beside the lexical launch config so startup finds the sole pending authority before trusting candidate contents:

Assign a distinct runtime_state_dir to every concurrently running rustbgpd daemon. The directory is single-writer runtime state; sharing it between live processes is unsupported even when their configuration files differ.

Stop the daemon before deleting blackhole-owned.json; deletion intentionally preserves all surviving marker rows as foreign. Never share runtime_state_dir between live instances, and do not downgrade to marker-only adoption without a drain plan.

The lexical config, metadata, and raw-prior paths resolve to absolute identities. A writer or present pending object requires daemon-owned real parents that are not group- or world-writable; locator absence carries no authority or ordinary-startup storage requirement. Pending and staging files must be daemon-owned regular files with mode 0600; symlinks and special files fail closed. Locator unlink plus parent fsync is terminal; only subsequent verified metadata/raw cleanup and pending-directory fsync are warning-only.

Production reads and writes v3. Retired v1/v2 authority refuses boot untouched; finish it before upgrade or recover with rustbgpd v0.64.0. Offline --check does not inspect this runtime state. Before downgrade, finish or abort v3 and never delete its live locator manually.

Routing state is not restored. The optional shutdown checkpoint contains only eligible post-import-policy Adj-RIB-In views for future use; Loc-RIB, Adj-RIB-Out, and policy evaluation state are not checkpointed, and the current startup path loads none of it. Routing state rebuilds from peer routes after restart. GR and checkpoint state support bounded control-plane restart handling, but do not guarantee forwarding continuity; peers may withdraw or replace routes, and therefore forwarding, until normal convergence.

The config file itself is mutable across runs: neighbor add/delete operations via gRPC persist back to the config file (see CONFIGURATION.md → "Config Persistence").

Sample profiles

The repo ships ten config profiles, plus a Docker Compose quick-start, under examples/ covering the standard deployment shapes. Pick the closest match, copy, edit:

Each profile validates with rustbgpd --check out of the box; edit the AS, addresses, and TLS material before deploying.

Security checklist

See SECURITY.md for the full posture document. The short version for first deployment:

• Bind addresses. prometheus_addr, grpc_tcp.address, grpc_uds.path default to listening on what the config says. Don't expose the Prometheus or gRPC endpoint to untrusted networks without authentication.
• gRPC. Use mTLS in production. Set [security.grpc] enforcement = "tier" and map principals to roles under [security.grpc.roles] (observer / automation / operator) per SECURITY.md; the per-method tier matrix itself is compiled into the daemon (crates/api/src/authz.rs), not configured per-tier in TOML. UDS with a restrictive mode is fine for single-host operator access.
• BGP authentication. TCP-MD5 (RFC 2385) and TCP-AO (RFC 5925) are both supported. TCP-AO is preferred and needs mainline Linux 6.7 or newer, or a downstream kernel with TCP-AO backported, built with CONFIG_TCP_AO=y; see ADR-0062.
• RPKI cache transport. RTR (RFC 8210) sessions are TCP with optional per-cache md5_password or tcp_ao authentication; they are never encrypted. Set a key when the cache supports one; otherwise run caches on loopback or a trusted segment, or tunnel the session (WireGuard, an SSH or stunnel forward). A remote cache reached over an untrusted path is a validation-integrity problem: anything on the path can rewrite the VRPs the daemon validates against.
• Firewall. Allow inbound TCP/179 only from configured peer addresses (or address ranges if using [[dynamic_neighbors]]). Block everything else.

Troubleshooting

For deeper investigation, raise the daemon log level globally with the RUST_LOG environment variable — there is no log-level key in [global.telemetry]; log_format selects "json" or "text" output and does not control verbosity. Under systemd, set the log level in the unit's [Service] section:

[Service]
Environment=RUST_LOG=debug

RUST_LOG accepts the usual tracing filter syntax, so a targeted directive beats a daemon-wide debug — including the per-peer span form documented in OPERATIONS.md:

Environment=RUST_LOG=info,[peer{peer_addr=10.0.0.1}]=debug

Per-peer verbosity also comes from [[neighbors]] log_level = "debug". Restart required for the global setting; per-peer log_level is live — re-applied on SIGHUP via a tracing reload handle that rebuilds the filter (base level plus every per-peer directive), so a level edit takes effect without a restart (see the reload matrix).

Related

• OPERATIONS.md — start/stop, reload, upgrade, debugging.
• CONFIGURATION.md — config reference.
• reload-matrix.md — per-field reload classification.
• SECURITY.md — security posture + firewall guidance.
• INTEROP.md — M-series interop matrix.
• adr/ — architectural decisions per protocol and design choice.