rustbgpd
Cookbook

EVPN fabric route reflector (control-plane only)

This recipe builds a control-plane-only VXLAN-EVPN fabric route reflector.

This recipe builds a control-plane-only VXLAN-EVPN fabric route reflector.

When this is you: You operate a VXLAN-EVPN leaf/spine fabric. Its VTEPs (FRR, SR Linux, GoBGP, or rustbgpd leaves) do the dataplane, and you want a lean, API-first reflector distributing the RFC 7432 routes between them. Scope, stated up front: this recipe is the RR role. The RR holds no EVI state, learns no MACs, and forwards no packets — it reflects EVPN Types 1–6 between clients, including Type 6 SMET relay. rustbgpd also has a bidirectional VTEP mode (alpha, Linux/VXLAN-only, with IRB and multi-homing — see evpn-enablement.md); that is a different deployment and not this document.

Proven by: M29 (EVPN RR capability + ListEvpnRoutes vs FRR), M30 (Type 2 MAC reflection end-to-end between kernel VXLAN VTEPs), M31 (MAC mobility + sticky preservation), M32 (multi-homing Type 1 EAD / Type 4 ES reflection), M82 (VLAN-aware-bundle reflection with non-zero Ethernet Tags — including rustbgpd's first vendor-NOS leg, Nokia SR Linux 25.10), and the M33 scale gate (50k reflected Type 2 routes + 60 s of 1,000-rps churn). The config below is a trimmed form of examples/rr-evpn-fabric/config.toml, which also disables [policy.explain] explicitly and uses the implicit gRPC socket path.

Type 6 SMET relay remains alpha. Its M113 raw-peer proof checks reflected wire bytes with an independent TShark decoder, including withdrawal and error recovery. The earlier receipts above cover other route types. M113 does not establish vendor interoperability; no SMET origination, IGMP/MLD proxy, or multicast forwarding is implemented. See the SMET boundary.

Types 1–5 reflection is the scoped RR-only route-reflector-evpn v1 role. Its compatibility promise covers the per-peer views rbgp evpn received PEER --route-type N and rbgp evpn advertised PEER --route-type N for N in 1–5, their --json page, and the named RIB, loop, discard and restart metrics; see the EVPN reflector boundary. The whole-table rbgp evpn listings under Verify and the EVPN event stream remain useful but alpha.

Config

[global]
asn = 65000
router_id = "10.0.0.100"
listen_port = 179
cluster_id = "10.0.0.100"
ebgp_requires_policy = false

[global.telemetry]
prometheus_addr = "127.0.0.1:9179"
log_format = "json"

# Owner-only local socket (default mode 0600): clients are authorized as the
# implicit "local-operator" principal — no [security.grpc] block needed.
[global.telemetry.grpc_uds]
path = "/var/lib/rustbgpd/grpc.sock"

# All VTEPs are iBGP RR clients on the l2vpn_evpn family. Empty
# [[evpn_instances]] (none configured) selects pure RR mode: no local
# EVI state, no kernel programming, reflection only.
[[neighbors]]
address = "10.0.0.1"
remote_asn = 65000
description = "vtep-leaf-01"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = true

[[neighbors]]
address = "10.0.0.2"
remote_asn = 65000
description = "vtep-leaf-02"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = true

[[neighbors]]
address = "10.0.0.3"
remote_asn = 65000
description = "vtep-leaf-03"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = true

For larger fabrics, template the leaves with a peer group and an auto-accept range over the VTEP loopback subnet — the unicast RR recipe shows the [peer_groups] + [[dynamic_neighbors]] shape; swap the families for ["l2vpn_evpn"].

Verify

$ export RUSTBGPD_ADDR=unix:///var/lib/rustbgpd/grpc.sock
$ rbgp neighbor                      # every leaf Established, l2vpn_evpn negotiated
$ rbgp evpn                          # all reflected EVPN routes
$ rbgp evpn --route-type 2           # MAC/IP advertisements only
$ rbgp evpn --route-type 3           # IMET (BUM flooding) entries
$ rbgp evpn --rd 65000:100           # one EVI's view
$ rbgp evpn --neighbor 10.0.0.1      # what leaf-01 contributed

Expected shape: one row per (RD, route-type, key) with the originating peer; after two leaves come up with the same L2VNI you should see each leaf's Type 3 IMET route reflected to the other, then Type 2 rows appearing as MACs are learned.

Two things that look odd but are correct:

  • The same MAC under two Ethernet Tags is two routes. In VLAN-aware-bundle service the non-zero Ethernet Tag is part of the route identity (ADR-0092); the RR keys and reflects them separately, tag-verbatim, and a single-tag withdraw removes exactly that entry (M82).
  • Attributes pass through untouched. RD, label/VNI, next-hop, RTs, MAC-mobility sequence numbers, ESI — the RR adds ORIGINATOR_ID and CLUSTER_LIST per RFC 4456 and changes nothing else. DF election, ARP/ND suppression, and mobility sequencing are the VTEPs' business.

Watch

The Grafana overview session and RIB-scale rows apply as-is (bgp_session_state_transitions_total, bgp_rib_outbound_registered_peers, update-group gauges). Per-VNI EVPN metric families (evpn_*) are VTEP-mode surface and stay empty in the RR role — deliberately not on the overview dashboard.

Size the RR's memory from receivers × reflected EVPN routes. Each client has its own EVPN Adj-RIB-Out, at roughly 250–500 B per route per receiver depending on table fill: 150 clients × 37,250 reflected routes held a flat live heap of about 2.1 GB in the EVPN RR soak. The unicast RR figures do not transfer.

Route churn is the fabric health signal on an EVPN RR. EVPN route events have their own category, which a default event stream does not include: watch rbgp events watch --category evpn (or the EVPN history, rbgp events evpn --limit 200, filterable with --route-type 2, --rd, and --neighbor) during rollouts. MAC-mobility wars show up as a tight add/withdraw loop on one MAC key with a climbing sequence number.

Failure modes

A leaf's routes aren't reaching another leaf. Start with rbgp evpn received 10.0.0.1 for accepted post-policy source routes and rbgp evpn advertised 10.0.0.2 for committed destination output. Both commands accept type/RD filters and return a bounded page; use the returned --page-token to continue. Then explain the exact key with its source and destination. For a MAC-only Type 2 route:

rbgp evpn explain mac-ip --rd 65000:100 --mac 02:00:00:00:00:11 \
  --received-from 10.0.0.1 --advertised-to 10.0.0.2

Add --ip for a MAC+IP key; use the route's actual --ethernet-tag when it is nonzero. Other route types have their own exact selectors. The result separates retained accepted input, installed best, fresh selection, current export gates, and committed output. A missing accepted source does not prove the leaf never sent the route, and committed output does not prove the other leaf received it. Check selection_deferred and outbound_dirty before expecting installed or committed state to match a current comparison.

To inspect SMET relay, filter on Type 6 and explain its exact key:

rbgp evpn received 10.0.0.1 --route-type 6
rbgp evpn explain smet --rd 65000:100 --source '*' --group 239.1.2.3 \
  --originator-ip 2001:db8::1 --advertised-to 10.0.0.2

'*' is the route's literal wildcard source, not a query pattern. Source/group families must agree when both are concrete; the originator family is independent.

Two common reflection stops are family — the destination did not negotiate l2vpn_evpn — and RR rules — the source and destination's route_reflector_client settings do not permit this reflection. The trace shows the stopping gate. An FRR leaf missing neighbor X activate under address-family l2vpn evpn can establish a session while receiving no EVPN routes, as demonstrated by the M29 fixture.

A third stop, rt_membership, applies when the destination leaf also negotiated rtc: it receives only EVPN routes with a Route Target inside the RT membership it advertised (Type 4 routes match on their ES-Import RT), and nothing until it advertises some. rbgp rib rtc --neighbor <leaf> lists the membership that leaf sent; the default (zero-length) membership restores the unfiltered feed.

Vendor NOS quirks. From the M82 SR Linux leg: SR Linux enforces one EVI per mac-vrf (bundle identity = shared RT + Ethernet Tag, per-BD RDs) and needs an explicit transport local-address on the BGP group, or it sources the session from its system0 address and the RR's neighbor stanza never matches. Recorded in upstream-findings.md and the M82 lab fixtures.

Reflected state survives leaf restarts? Add GR to the neighbor stanzas (graceful_restart = true and friends, as in the unicast recipe) if your leaves support GR for EVPN; without it, a leaf bounce withdraws its routes fabric-wide and they re-learn on re-establish.

Source on GitHub

On this page