doublezerod: add periodic kernel route reconciliation

[nikw9944]

Resolves: #3669

Summary of Changes

• Add periodic kernel route reconciliation to doublezerod: BGP routes deleted from the kernel by another process or an administrator are detected and reinstalled on a configurable tick (-route-reconcile-interval, default 30s; 0 disables)
• Reconciliation is independent of the route-liveness subsystem: it is a transparent routing.Netlinker decorator (internal/reconcile) at the base of the routing chain, just above raw netlink. Every kernel route write bottoms out at this layer, so it works with liveness in passive or active mode and equally with liveness disabled — a tenant opting out of route liveness can still use route reconciliation
• Expose doublezero_route_reconcile_reinstalls_total and doublezero_route_reconcile_failures_total. The route_reconcile naming (flag, metrics, log prefix) is deliberately distinct from the existing onchain services reconciler (-reconciler-poll-interval, doublezero_reconciler_*) to avoid confusion
• Lifecycle correctness:
  • RouteDelete untracks under the lock before the kernel delete, and the reinstall re-checks the tracked set under the same lock, closing the withdraw/reconcile resurrection race on every withdrawal path
  • Untracking is protocol-agnostic: BGP withdraw-driven deletes are constructed without a Protocol, so a protocol filter would leak entries and resurrect withdrawn routes
  • TunnelDelete purges tracked routes via the tunnel's remote-overlay nexthop: NoUninstall teardowns (IBRL-with-allocated-IP) never issue a RouteDelete, and the kernel drops routes with their link — without the purge the reconciler would try to reinstall onto a dead tunnel forever
  • The decorator sits below ConfiguredRouteReaderWriter, so excluded destinations never reach it and are never tracked or falsely reinstalled
• Route matching uses the full destination prefix (Dst.String()) plus normalized source/nexthop IPs, so a 10.0.0.0/16 kernel route does not satisfy a tracked 10.0.0.0/24, and 4-byte vs 16-byte net.IP forms compare equal
• Promote "session down (passive; keeping route)" log messages from Debug to Info, otherwise it's possible to log multiple 'liveness: session up' messages in a row

Design history

Earlier revisions implemented reconciliation inside the liveness manager (keyed off its installed[] map); review feedback on that design (kill switch, excluded-route churn, TOCTOU ordering, prefix matching, main-table filtering) is preserved in the review threads and all of it carries over into the decorator implementation. The restructure was prompted by the manager-based design making reconciliation unusable when route liveness is disabled.

Testing Verification

• internal/reconcile unit tests cover: reinstalling a missing route (+ reinstall metric), skipping a route present in the kernel (kernel echo is a freshly-constructed route in 16-byte IP form, exercising key normalization rather than pointer identity), prefix-mismatch reinstall (/24 kernel route does not satisfy tracked /32), protocol-agnostic untrack on withdraw-shaped deletes, withdraw-during-reconcile not resurrected, non-BGP routes untracked, failure metric increments and the route stays tracked for retry, TunnelDelete purging only that tunnel's routes, excluded destinations never tracked (verified through a real ConfiguredRouteReaderWriter layered above the reconciler), and 0 interval disabling the ticker
• Liveness manager tests updated: reconciliation tests moved to the new package; the passive WithdrawRoute clear-before-delete ordering test remains
• RouteByProtocol's main-table-only behavior is documented on the method and noted at the reconcile call site

[nikw9944]

Route Reconciliation Performance Analysis

reconcileRoutes() runs every 30s with 4 phases:

1. Lock → snapshot → unlock — iterates installed ∩ desired into a local slice
2. Netlink dump (no lock held) — RouteListFiltered(FAMILY_V4, RTPROT_BGP) dumps all kernel BGP routes
3. Build hash set + diff — builds map[kernelKey]struct{} from kernel routes, checks each installed route against it
4. Reinstall missing — brief re-lock per missing route, then RouteAdd

CPU cost per reconciliation cycle

Routes	Lock hold (step 1)	Netlink dump (step 2)	Map build + diff (step 3)	Total per cycle	Amortized over 30s
100	~1 μs	~0.5 ms	~50 μs	~0.6 ms	0.002%
1,000	~5 μs	~3 ms	~0.5 ms	~4 ms	0.013%
1,000,000	~5 ms	~1-2 s	~300-500 ms	~2-3 s	~7-10%

Estimation methodology

Lock hold (step 1): Map iteration over installed checking membership in desired, appending matches to a slice. Go map iteration is ~10ns/entry. 100 entries × 10ns = 1μs; 1000 × 10ns = 10μs (rounded to ~5μs accounting for the append being fast). At 1M entries, iteration + slice growth + memory allocation dominates: ~5ms.

Netlink dump (step 2): RouteListFiltered issues a single RTM_GETROUTE netlink dump; the kernel filters by protocol server-side and streams matching routes back. Each route message is ~100-200 bytes of netlink payload. At 100 routes that's ~10-20KB parsed by vishvananda/netlink — well under 1ms. At 1,000 routes, ~100-200KB, ~3ms. At 1M routes, ~100-200MB of netlink data to receive and deserialize — estimated 1-2s based on netlink socket throughput (~100-200MB/s for dump operations).

Map build + diff (step 3): For each kernel route, we call IP.To4().String() on 3 fields (dst, nexthop, src) creating heap-allocated strings, then insert into map[kernelKey]struct{}. String conversion + map insert is ~200-500ns/route. At 100 routes: ~50μs. At 1,000: ~500μs. At 1M: ~300-500ms (dominated by allocations and GC pressure from ~3M short-lived strings). The diff itself is O(installed) map lookups at O(1) each — negligible relative to the build.

Amortized CPU: total_per_cycle / 30s. E.g., at 1,000 routes: 4ms / 30,000ms = 0.013%.

Lock contention with HandleRx

The lock is not held during the expensive netlink syscall (step 2). The snapshot in step 1 holds m.mu only during map iteration. HandleRx (the hot path) also holds m.mu for its duration, so these can contend — but at 100-1000 routes, a 1-5μs delay is negligible compared to the 50ms-3s liveness TX intervals. At 1M routes the ~5ms lock hold becomes measurable but is still small relative to liveness timers.

Practical impact on doublezerod CPU usage

Given a ~3% baseline CPU on a modern x86 core, this change adds effectively zero overhead at realistic route counts (low hundreds). The 1M route case is pathological for a doublezerod client and would have other scaling bottlenecks (BGP convergence, session state memory, netlink install throughput) long before reconciliation matters.

[ben-dz]

1. The 0 disables kill switch doesn't work — Validate() rewrites 0 to the 30s default, so there's no way to turn this new periodic dataplane writer off during staged rollout.
2. Excluded routes churn forever — on any host using a route exclude list, the reinstall counter and a Warn log fire every tick, permanently, defeating the metric's purpose.

Details inline.

PR-description corrections (not line-anchored):

• The description claims a unit test for "incrementing the install failure metric on RouteAdd error". No such test exists — the PR adds three (ReinstallsMissing, SkipsPresent, SkipsUninstalled). Either add it (a few lines with the existing mock) or drop the claim.
• It claims to add doublezero_liveness_route_install_failures_total. That metric already existed; this PR only adds a new increment site (the new metric is doublezero_liveness_route_reinstalls_total).
• "Prevent TOCTOU race" overstates it — see the inline comment; the race is narrowed, not closed.

[nikw9944]

@ben-dz thanks for the thorough review — all points addressed in 789f442, with per-comment replies inline. Summary:

Should-fix (inline):

• Kill switch — Validate() no longer rewrites 0; the > 0 guard now genuinely disables reconciliation, and the duplicate default constant is gone.
• Excluded routes — reconcileRoutes skips m.cr.IsExcluded(...) destinations, so they no longer churn the counter/logs.
• TOCTOU — m.mu is now held across the re-check and RouteAdd, closing the race (not just narrowing it).
• Prefix match — kernel-set key uses the full *net.IPNet.String() prefix on both sides.
• RouteByProtocol main-table assumption — documented with a NOTE and a pointer to add RT_FILTER_TABLE if ever used for tables 100/101.

PR-description corrections:

• Added TestClient_Liveness_Manager_ReconcileRoutes_IncrementsInstallFailureMetric, so the install-failure-on-RouteAdd-error test claim is now real.
• Description corrected: it now states the PR increments the existing doublezero_liveness_route_install_failures_total and adds doublezero_liveness_route_reinstalls_total.
• "Prevent TOCTOU" reworded to "Close … by holding m.mu", which is now accurate given the lock is held across the syscall.

[ben-dz]

Post-fix review at af90168. The six fixes from the prior review round all hold up, and production correctness is sound for the IBRL case (verified Table=254/RT_TABLE_MAIN, Protocol=RTPROT_BGP, explicit Src round-trip; RouteAdd is idempotent RouteReplace). No critical/high issues. Three findings worth attention: (M1) the resurrection race is closed for onSessionDown but a symmetric narrow window remains on the passive-mode WithdrawRoute path, which deletes the kernel route before clearing installed — so the PR's "Prevent TOCTOU race" claim is accurate only for the onSessionDown ordering; (M2) the new reconcile tests are partly fictional — they use table 100 (production is 254, which RouteByProtocol filters out) and the SkipsPresent test returns the identical *Route pointer, so kernel-vs-desired key matching is never genuinely exercised; (L1) the RouteAdd syscall is run while holding m.mu, a deviation from the file's no-lock-across-syscall convention (a documented, reviewer-endorsed tradeoff).

Findings not anchored to the current diff:

• client/doublezerod/internal/liveness/manager.go:451 — medium: Resurrection race still open on the passive-mode WithdrawRoute path. This branch issues RouteDelete (line 451) before clearing installed[rk] (line 463) — the opposite ordering from onSessionDown (clears at :838, deletes at :890), which is the ordering the TOCTOU fix depends on. If reconcile snapshots the route as installed, then passive WithdrawRoute runs its RouteDelete here but is preempted before acquiring the lock at :459, reconcile's kernel query (:945) sees the route missing, takes the lock at :991 with installed[rk] still true, and re-adds the route. WithdrawRoute then clears the maps, leaving a permanent stale kernel route the manager believes is withdrawn. Likelihood is low but the consequence is a stale dataplane route to a withdrawn destination. Fix: clear installed[rk]/desired[rk] under the lock before RouteDelete here, mirroring onSessionDown. The PR's "Prevent TOCTOU race" claim is accurate only for the onSessionDown ordering.

[ben-dz]

Reconcile tests don't reproduce production kernel filtering or representation. newTestRoute (main_test.go:71) defaults to Table:100, but liveness runs only in IBRL mode (RT_TABLE_MAIN=254, services/ibrl.go:64), and RouteByProtocol sets only RT_FILTER_PROTOCOL so it returns only main-table routes — a table-100 route would never come back from the real backend. SkipsPresent passes only because the mock returns the identical &r.Route pointer the manager installed, guaranteeing a key match regardless of table and bypassing real representation differences (4-byte vs 16-byte net.IP, prefsrc echo, mask normalization). Recommend using Table:254 and having the mock return a freshly-constructed *routing.Route with the same field values so key construction is genuinely exercised on both sides.

[nikw9944]

Fixed in ecca28b. newTestRoute now defaults to Table: unix.RT_TABLE_MAIN (254) so the reconcile tests use the same table liveness actually attaches to in IBRL mode and that RouteByProtocol returns. SkipsPresent now returns a freshly-constructed *routing.Route with the same field values instead of the identical &r.Route pointer, so kernelKey construction is genuinely exercised on both the kernel and desired sides (4-byte vs 16-byte net.IP, prefsrc echo, mask normalization) rather than matching by pointer identity.

Changes Addressed.

[nikw9944]

Addressed the post-fix review findings in daddc70:

M1 — passive-mode WithdrawRoute resurrection race (now closed). The passive path issued RouteDelete before clearing installed[rk], the opposite of onSessionDown's clear-then-delete ordering that the reconcile fix depends on. Restructured WithdrawRoute to clear desired[rk]/installed[rk] under m.mu first and then issue a single kernel RouteDelete (passive: always; active: only when previously installed). Reconcile now either observes the cleared flag at its re-check and skips, or completes its add before the delete lands. The race is closed on both withdrawal paths, so the "Prevent TOCTOU race" claim now holds for the passive path too. Added TestClient_Liveness_Manager_WithdrawRoute_PassiveClearsInstalledBeforeDelete asserting the delete sees installed[rk] == false.

M2 — reconcile tests now reflect production. Changed the default test-route table to RT_TABLE_MAIN (since RouteByProtocol only sets RT_FILTER_PROTOCOL and returns main-table routes, a table-100 route would never come back from the real backend), and SkipsPresent now returns a freshly-constructed *routing.Route with the same field values instead of the identical &r.Route pointer, so kernelKey construction is genuinely exercised on both the kernel and desired sides.

L1 — RouteAdd under m.mu. Left as-is; it's the deliberate, reviewer-endorsed tradeoff that makes M1's closure possible, and the syscall under the lock only happens for genuinely-missing routes (rare by definition). Documented inline.

[nikw9944]

Addressed the post-fix review (M1/M2/L1) in ecca28b, and rebased the branch onto latest main.

• M2 — reconcile tests didn't reflect production (thread on manager_test.go): newTestRoute now defaults to RT_TABLE_MAIN, and SkipsPresent returns a freshly-constructed *routing.Route instead of the installed pointer, so kernelKey matching is genuinely exercised on both sides. Also added TestClient_Liveness_Manager_WithdrawRoute_PassiveClearsInstalledBeforeDelete.
• M1 — resurrection race on the passive-mode WithdrawRoute path (raised in the review body, not line-anchored): WithdrawRoute now clears desired[rk]/installed[rk] under m.mu before issuing the kernel RouteDelete, mirroring onSessionDown. Reconcile therefore either observes the cleared flag at its re-check and skips, or completes its add before the delete lands. The race is now closed on both withdrawal paths, so the "Prevent TOCTOU race" claim holds for the passive path too.
• L1 — RouteAdd under m.mu: left as-is. It's the deliberate, reviewer-endorsed tradeoff that makes M1's closure possible, and the syscall under the lock only runs for genuinely-missing routes (rare by definition). Documented inline.

The six fixes from the prior round are unchanged and intact through the rebase. Full liveness and routing suites pass locally.

[nikw9944]

Restructured in 4a4ffc9: route reconciliation is now fully independent of the route-liveness subsystem.

Why. In the prior revisions reconciliation lived inside the liveness manager, so disabling liveness (-route-liveness-enable-passive=false -route-liveness-enable-active=false) also removed reconciliation — a tenant opting out of route liveness was prevented from using it.

How. New internal/reconcile package: a transparent routing.Netlinker decorator at the base of the routing chain, just above raw netlink. It records every BGP route actually pushed to the kernel and reinstalls any that go missing on a tick. Since every kernel route write bottoms out at this layer, it works identically with liveness passive, active, or fully disabled. The flag is renamed to -route-reconcile-interval (no longer a liveness knob) and the metrics to doublezero_route_reconcile_{reinstalls,failures}_total — deliberately distinct from the onchain services reconciler's -reconciler-poll-interval / doublezero_reconciler_* namespace.

Prior review feedback carries over structurally:

• Kill switch: 0 disables the ticker; the default lives only in the main.go flag default.
• Excluded churn: the decorator sits below ConfiguredRouteReaderWriter, so excluded destinations never reach it — nothing to skip, by construction.
• Resurrection race: RouteDelete untracks under the lock before the kernel delete; the reinstall re-checks the tracked set under the same lock. This now covers every withdrawal path uniformly (the manager-based fix needed per-path ordering audits).
• Prefix + representation matching: full Dst.String() prefix, IPs normalized to 4-byte form; tests use freshly-constructed kernel echoes, not shared pointers.
• Main-table NOTE on RouteByProtocol retained and referenced at the reconcile call site.

New lifecycle handling the decorator position surfaced:

• BGP withdraw-driven deletes are constructed without Protocol, so untracking is protocol-agnostic (a BGP-only filter would leak entries and resurrect withdrawn routes).
• TunnelDelete purges tracked routes via the tunnel's remote-overlay nexthop: NoUninstall teardowns (IBRL-with-allocated-IP) never issue a RouteDelete and the kernel drops routes with their link; without the purge the reconciler would attempt reinstalls onto a dead tunnel forever.

Ten unit tests in the new package cover all of the above; the liveness manager's reconcile code, metric, and tests are removed (the passive WithdrawRoute clear-before-delete ordering test remains). Full doublezerod suite passes locally; golangci-lint clean.