flock-agent: GC orphaned allocations; retry birdc on socket-not-ready
flock PR validation / validate (pull_request) Successful in 11s
flock PR validation / validate (pull_request) Successful in 11s
Three defects enabled the 2026-08-16 Gitea blackhole (bug-wdgjpz3a00gd): 1. Orphaned allocation GC missing: ungraceful eviction (TaintManagerEviction) never calls CNI DEL, so the old node keeps advertising the pod's public /128 via BGP. Older allocation wins BGP path selection; live pod's node yields → blackhole. Fix: after the pod informer syncs at startup, sweep all committed allocations via orphanedCommitted(). Any allocation whose owner pod is absent from the node (or whose UID mismatches, indicating name reuse) is torn down, removed from the store, and released from IPAM. A 60 s periodic GC goroutine provides the same sweep while the agent runs. 2. renderBird outside-aggregate IP loop lacked pod liveness check: stale committed allocations caused BIRD to keep advertising the /128 even in steady state between GC ticks. Fix: before adding an outside-aggregate primary IP to the BIRD export, verify the pod is still in the node-scoped informer cache with a matching UID. Orphans are skipped silently; the GC cleans them on the next tick. 3. birdc startup race: the agent's first Render() fires before BIRD has bound /run/flock/bird.ctl, so the configure call silently fails with "Unable to connect" and the initial routes are never advertised. A container-only flock-agent restart (BIRD left running) avoids the race; a full pod restart re-hits it. Fix: reload() now retries up to 20 × 500 ms on socket-absent and "Unable to connect" conditions. Any other birdc failure (syntax error, etc.) is not retried. Fixes bug-wdgjpz3a00gd Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+40
-8
@@ -8,6 +8,7 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -137,6 +138,15 @@ func (b *BirdManager) scheduleReload() {
|
||||
})
|
||||
}
|
||||
|
||||
// birdcMaxAttempts and birdcRetryDelay bound the startup-socket retry loop.
|
||||
// BIRD may not have bound its control socket yet when flock-agent first
|
||||
// tries to configure it; 20 × 500 ms = 10 s covers the typical BIRD
|
||||
// startup window without blocking indefinitely.
|
||||
const (
|
||||
birdcMaxAttempts = 20
|
||||
birdcRetryDelay = 500 * time.Millisecond
|
||||
)
|
||||
|
||||
func (b *BirdManager) reload() {
|
||||
birdctl := b.BirdctlPath
|
||||
if birdctl == "" {
|
||||
@@ -146,18 +156,40 @@ func (b *BirdManager) reload() {
|
||||
if socket == "" {
|
||||
socket = "/run/flock/bird.ctl"
|
||||
}
|
||||
cmd := exec.Command(birdctl, "-s", socket, "configure")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
// First-run case: bird may not be ready yet — retry on next change.
|
||||
if errors.Is(err, exec.ErrNotFound) || os.IsNotExist(err) {
|
||||
b.Logger.Warn("birdc not available", "err", err)
|
||||
|
||||
for attempt := 1; attempt <= birdcMaxAttempts; attempt++ {
|
||||
if attempt > 1 {
|
||||
time.Sleep(birdcRetryDelay)
|
||||
}
|
||||
// Socket absent → BIRD hasn't bound it yet; retry.
|
||||
if _, err := os.Stat(socket); os.IsNotExist(err) {
|
||||
b.Logger.Debug("birdc socket not ready, retrying",
|
||||
"attempt", attempt, "socket", socket)
|
||||
continue
|
||||
}
|
||||
cmd := exec.Command(birdctl, "-s", socket, "configure")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err == nil {
|
||||
b.Logger.Info("birdc configure ok", "out", string(out))
|
||||
return
|
||||
}
|
||||
b.Logger.Warn("birdc reload failed", "err", err, "out", string(out))
|
||||
if errors.Is(err, exec.ErrNotFound) {
|
||||
b.Logger.Warn("birdc not found", "err", err)
|
||||
return
|
||||
}
|
||||
outStr := string(out)
|
||||
// "Unable to connect" means BIRD exists but isn't listening yet.
|
||||
if strings.Contains(outStr, "Unable to connect") {
|
||||
b.Logger.Debug("birdc not ready, retrying",
|
||||
"attempt", attempt, "err", err)
|
||||
continue
|
||||
}
|
||||
// Any other failure (syntax error, etc.) is not retriable.
|
||||
b.Logger.Warn("birdc reload failed", "err", err, "out", outStr)
|
||||
return
|
||||
}
|
||||
b.Logger.Info("birdc configure ok", "out", string(out))
|
||||
b.Logger.Error("birdc configure gave up after retries",
|
||||
"socket", socket, "attempts", birdcMaxAttempts)
|
||||
}
|
||||
|
||||
// SummaryRoutes installs blackhole kernel routes for each NodeConfig CIDR.
|
||||
|
||||
Reference in New Issue
Block a user