EVPN fabric route reflector (control-plane only)
This recipe builds a control-plane-only VXLAN-EVPN fabric route reflector.
This recipe builds a control-plane-only VXLAN-EVPN fabric route reflector.
When this is you: You operate a VXLAN-EVPN leaf/spine fabric. Its
VTEPs (FRR, SR Linux, GoBGP, or rustbgpd leaves) do the dataplane, and
you want a lean, API-first reflector distributing the RFC 7432 routes
between them. Scope, stated up front: this recipe is the RR role. The
RR holds no EVI state, learns no MACs, and forwards no packets — it
reflects EVPN Types 1–6 between clients, including Type 6 SMET relay. rustbgpd
also has a bidirectional VTEP mode (alpha, Linux/VXLAN-only, with IRB
and multi-homing — see
evpn-enablement.md); that is a different
deployment and not this document.
Proven by: M29
(EVPN RR capability + ListEvpnRoutes vs FRR), M30 (Type 2 MAC
reflection end-to-end between kernel VXLAN VTEPs), M31 (MAC mobility +
sticky preservation), M32 (multi-homing Type 1 EAD / Type 4 ES
reflection), M82 (VLAN-aware-bundle reflection with non-zero Ethernet
Tags — including rustbgpd's first vendor-NOS leg, Nokia SR Linux
25.10), and the M33 scale gate (50k reflected Type 2 routes + 60 s of
1,000-rps churn). The config below is a trimmed form of
examples/rr-evpn-fabric/config.toml,
which also disables [policy.explain] explicitly and uses the implicit gRPC
socket path.
Type 6 SMET relay remains alpha. Its M113 raw-peer proof checks reflected wire bytes with an independent TShark decoder, including withdrawal and error recovery. The earlier receipts above cover other route types. M113 does not establish vendor interoperability; no SMET origination, IGMP/MLD proxy, or multicast forwarding is implemented. See the SMET boundary.
Types 1–5 reflection is the scoped RR-only route-reflector-evpn v1 role.
Its compatibility promise covers the per-peer views
rbgp evpn received PEER --route-type N and
rbgp evpn advertised PEER --route-type N for N in 1–5, their --json page,
and the named RIB, loop, discard and restart metrics; see the
EVPN reflector boundary.
The whole-table rbgp evpn listings under Verify and the EVPN event stream
remain useful but alpha.
Config
[global]
asn = 65000
router_id = "10.0.0.100"
listen_port = 179
cluster_id = "10.0.0.100"
ebgp_requires_policy = false
[global.telemetry]
prometheus_addr = "127.0.0.1:9179"
log_format = "json"
# Owner-only local socket (default mode 0600): clients are authorized as the
# implicit "local-operator" principal — no [security.grpc] block needed.
[global.telemetry.grpc_uds]
path = "/var/lib/rustbgpd/grpc.sock"
# All VTEPs are iBGP RR clients on the l2vpn_evpn family. Empty
# [[evpn_instances]] (none configured) selects pure RR mode: no local
# EVI state, no kernel programming, reflection only.
[[neighbors]]
address = "10.0.0.1"
remote_asn = 65000
description = "vtep-leaf-01"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = true
[[neighbors]]
address = "10.0.0.2"
remote_asn = 65000
description = "vtep-leaf-02"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = true
[[neighbors]]
address = "10.0.0.3"
remote_asn = 65000
description = "vtep-leaf-03"
hold_time = 180
families = ["l2vpn_evpn"]
route_reflector_client = trueFor larger fabrics, template the leaves with a peer group and an
auto-accept range over the VTEP loopback subnet — the
unicast RR recipe shows the
[peer_groups] + [[dynamic_neighbors]] shape; swap the families for
["l2vpn_evpn"].
Verify
$ export RUSTBGPD_ADDR=unix:///var/lib/rustbgpd/grpc.sock
$ rbgp neighbor # every leaf Established, l2vpn_evpn negotiated
$ rbgp evpn # all reflected EVPN routes
$ rbgp evpn --route-type 2 # MAC/IP advertisements only
$ rbgp evpn --route-type 3 # IMET (BUM flooding) entries
$ rbgp evpn --rd 65000:100 # one EVI's view
$ rbgp evpn --neighbor 10.0.0.1 # what leaf-01 contributedExpected shape: one row per (RD, route-type, key) with the originating peer; after two leaves come up with the same L2VNI you should see each leaf's Type 3 IMET route reflected to the other, then Type 2 rows appearing as MACs are learned.
Two things that look odd but are correct:
- The same MAC under two Ethernet Tags is two routes. In VLAN-aware-bundle service the non-zero Ethernet Tag is part of the route identity (ADR-0092); the RR keys and reflects them separately, tag-verbatim, and a single-tag withdraw removes exactly that entry (M82).
- Attributes pass through untouched. RD, label/VNI, next-hop, RTs, MAC-mobility sequence numbers, ESI — the RR adds ORIGINATOR_ID and CLUSTER_LIST per RFC 4456 and changes nothing else. DF election, ARP/ND suppression, and mobility sequencing are the VTEPs' business.
Watch
The Grafana overview session and RIB-scale rows apply
as-is (bgp_session_state_transitions_total,
bgp_rib_outbound_registered_peers, update-group gauges). Per-VNI
EVPN metric families (evpn_*) are VTEP-mode surface and stay empty
in the RR role — deliberately not on the overview dashboard.
Size the RR's memory from receivers × reflected EVPN routes. Each client has its own EVPN Adj-RIB-Out, at roughly 250–500 B per route per receiver depending on table fill: 150 clients × 37,250 reflected routes held a flat live heap of about 2.1 GB in the EVPN RR soak. The unicast RR figures do not transfer.
Route churn is the fabric health signal on an EVPN RR. EVPN route events
have their own category, which a default event stream does not include: watch
rbgp events watch --category evpn (or the EVPN history,
rbgp events evpn --limit 200, filterable with --route-type 2, --rd, and
--neighbor) during rollouts. MAC-mobility wars show up as a tight
add/withdraw loop on one MAC key with a climbing sequence number.
Failure modes
A leaf's routes aren't reaching another leaf. Start with
rbgp evpn received 10.0.0.1 for accepted post-policy source routes and
rbgp evpn advertised 10.0.0.2 for committed destination output. Both
commands accept type/RD filters and return a bounded page; use the returned
--page-token to continue. Then explain the exact key with its source and
destination. For a MAC-only Type 2 route:
rbgp evpn explain mac-ip --rd 65000:100 --mac 02:00:00:00:00:11 \
--received-from 10.0.0.1 --advertised-to 10.0.0.2Add --ip for a MAC+IP key; use the route's actual --ethernet-tag when it
is nonzero. Other route types have their own
exact selectors.
The result separates retained accepted input, installed best, fresh selection,
current export gates, and committed output. A missing accepted source does not
prove the leaf never sent the route, and committed output does not prove the
other leaf received it. Check selection_deferred and outbound_dirty before
expecting installed or committed state to match a current comparison.
To inspect SMET relay, filter on Type 6 and explain its exact key:
rbgp evpn received 10.0.0.1 --route-type 6
rbgp evpn explain smet --rd 65000:100 --source '*' --group 239.1.2.3 \
--originator-ip 2001:db8::1 --advertised-to 10.0.0.2'*' is the route's literal wildcard source, not a query pattern. Source/group
families must agree when both are concrete; the originator family is independent.
Two common reflection stops are family — the destination did not negotiate
l2vpn_evpn — and RR rules — the source and destination's
route_reflector_client settings do not permit this reflection. The trace
shows the stopping gate. An FRR leaf missing neighbor X activate under
address-family l2vpn evpn can establish a session while receiving no EVPN
routes, as demonstrated by the M29 fixture.
A third stop, rt_membership, applies when the destination leaf also
negotiated rtc: it receives only EVPN routes with a Route Target inside the
RT membership it advertised (Type 4 routes match on their ES-Import RT), and
nothing until it advertises some. rbgp rib rtc --neighbor <leaf> lists the
membership that leaf sent; the default (zero-length) membership restores the
unfiltered feed.
Vendor NOS quirks. From the M82 SR Linux leg: SR Linux enforces
one EVI per mac-vrf (bundle identity = shared RT + Ethernet Tag,
per-BD RDs) and needs an explicit transport local-address on the BGP
group, or it sources the session from its system0 address and the RR's
neighbor stanza never matches. Recorded in
upstream-findings.md and the M82 lab
fixtures.
Reflected state survives leaf restarts? Add GR to the neighbor
stanzas (graceful_restart = true and friends, as in the
unicast recipe) if your leaves support
GR for EVPN; without it, a leaf bounce withdraws its routes
fabric-wide and they re-learn on re-establish.