flock-agent: GC orphaned allocations; retry birdc on socket-not-ready
flock PR validation / validate (pull_request) Successful in 11s

Three defects enabled the 2026-08-16 Gitea blackhole (bug-wdgjpz3a00gd):

1. Orphaned allocation GC missing: ungraceful eviction (TaintManagerEviction)
   never calls CNI DEL, so the old node keeps advertising the pod's public
   /128 via BGP. Older allocation wins BGP path selection; live pod's node
   yields → blackhole.

   Fix: after the pod informer syncs at startup, sweep all committed
   allocations via orphanedCommitted(). Any allocation whose owner pod is
   absent from the node (or whose UID mismatches, indicating name reuse) is
   torn down, removed from the store, and released from IPAM. A 60 s
   periodic GC goroutine provides the same sweep while the agent runs.

2. renderBird outside-aggregate IP loop lacked pod liveness check: stale
   committed allocations caused BIRD to keep advertising the /128 even in
   steady state between GC ticks.

   Fix: before adding an outside-aggregate primary IP to the BIRD export,
   verify the pod is still in the node-scoped informer cache with a matching
   UID. Orphans are skipped silently; the GC cleans them on the next tick.

3. birdc startup race: the agent's first Render() fires before BIRD has
   bound /run/flock/bird.ctl, so the configure call silently fails with
   "Unable to connect" and the initial routes are never advertised. A
   container-only flock-agent restart (BIRD left running) avoids the race;
   a full pod restart re-hits it.

   Fix: reload() now retries up to 20 × 500 ms on socket-absent and
   "Unable to connect" conditions. Any other birdc failure (syntax error,
   etc.) is not retried.

Fixes bug-wdgjpz3a00gd

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
ops
2026-08-16 06:26:38 +00:00
co-authored by Claude Sonnet 4.6
parent 197bc6f3b8
commit c9950348ff
5 changed files with 247 additions and 8 deletions
+40 -8
View File
@@ -8,6 +8,7 @@ import (
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
@@ -137,6 +138,15 @@ func (b *BirdManager) scheduleReload() {
})
}
// birdcMaxAttempts and birdcRetryDelay bound the startup-socket retry loop.
// BIRD may not have bound its control socket yet when flock-agent first
// tries to configure it; 20 × 500 ms = 10 s covers the typical BIRD
// startup window without blocking indefinitely.
const (
birdcMaxAttempts = 20
birdcRetryDelay = 500 * time.Millisecond
)
func (b *BirdManager) reload() {
birdctl := b.BirdctlPath
if birdctl == "" {
@@ -146,18 +156,40 @@ func (b *BirdManager) reload() {
if socket == "" {
socket = "/run/flock/bird.ctl"
}
cmd := exec.Command(birdctl, "-s", socket, "configure")
out, err := cmd.CombinedOutput()
if err != nil {
// First-run case: bird may not be ready yet — retry on next change.
if errors.Is(err, exec.ErrNotFound) || os.IsNotExist(err) {
b.Logger.Warn("birdc not available", "err", err)
for attempt := 1; attempt <= birdcMaxAttempts; attempt++ {
if attempt > 1 {
time.Sleep(birdcRetryDelay)
}
// Socket absent → BIRD hasn't bound it yet; retry.
if _, err := os.Stat(socket); os.IsNotExist(err) {
b.Logger.Debug("birdc socket not ready, retrying",
"attempt", attempt, "socket", socket)
continue
}
cmd := exec.Command(birdctl, "-s", socket, "configure")
out, err := cmd.CombinedOutput()
if err == nil {
b.Logger.Info("birdc configure ok", "out", string(out))
return
}
b.Logger.Warn("birdc reload failed", "err", err, "out", string(out))
if errors.Is(err, exec.ErrNotFound) {
b.Logger.Warn("birdc not found", "err", err)
return
}
outStr := string(out)
// "Unable to connect" means BIRD exists but isn't listening yet.
if strings.Contains(outStr, "Unable to connect") {
b.Logger.Debug("birdc not ready, retrying",
"attempt", attempt, "err", err)
continue
}
// Any other failure (syntax error, etc.) is not retriable.
b.Logger.Warn("birdc reload failed", "err", err, "out", outStr)
return
}
b.Logger.Info("birdc configure ok", "out", string(out))
b.Logger.Error("birdc configure gave up after retries",
"socket", socket, "attempts", birdcMaxAttempts)
}
// SummaryRoutes installs blackhole kernel routes for each NodeConfig CIDR.