anycast: put IP on pod eth0, not lo
Build flock Image / build (push) Has been cancelled

The design doc's lo placement was motivated by avoiding NDP/ARP DAD
conflicts "across nodes advertising the same IP" — but flock pods each
sit on their own /64 veth subnet. DAD on eth0 only sees the host peer,
no cross-node L2.

With the IP on lo, the pod kernel doesn't reply to NDP solicits arriving
on eth0 (Linux default: answer NDP only for addresses on the receiving
interface). The host route `<ip>/128 dev flock<8hex>` causes the host
to do NDP for the destination on the veth; pod ignores; packet drops
silently between forwarding decision and transmit. Symptom: v4 anycast
works (proxy_arp=1 on the host veth handles ARP), v6 anycast doesn't.

Putting on eth0 makes NDP just work.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Donavan Fritz
2026-04-25 07:55:12 -05:00
parent 3f6dfd3e88
commit e1e9544e2e
+13 -11
View File
@@ -241,16 +241,18 @@ func configurePodSide(req SetupRequest) error {
} }
} }
// Anycast: assign each IP to pod lo. NOT on eth0 (avoids NDP/ARP // Anycast: assign each IP to pod eth0 (NOT lo).
// DAD conflicts when multiple replicas share the same IP). //
// The original design doc proposed lo to avoid NDP/ARP DAD
// conflicts "across nodes advertising the same IP". That concern
// doesn't apply to flock: each pod's veth is its own private /64,
// so DAD on eth0 only sees the veth peer (host) — no cross-node
// L2 contention. Putting the IP on eth0 instead means the pod
// kernel answers NDP solicits arriving on eth0 for that IP, which
// is what the host's /128 host route requires. With anycast on
// lo, NDP from the host side fails and the kernel drops the
// packet between routing decision and transmit.
if len(req.Anycast) > 0 { if len(req.Anycast) > 0 {
lo, err := netlink.LinkByName("lo")
if err != nil {
return fmt.Errorf("lookup pod lo: %w", err)
}
if err := netlink.LinkSetUp(lo); err != nil {
return fmt.Errorf("set up pod lo: %w", err)
}
for _, ip := range req.Anycast { for _, ip := range req.Anycast {
var mask net.IPMask var mask net.IPMask
if ip.To4() != nil { if ip.To4() != nil {
@@ -260,8 +262,8 @@ func configurePodSide(req SetupRequest) error {
mask = net.CIDRMask(128, 128) mask = net.CIDRMask(128, 128)
} }
a := &netlink.Addr{IPNet: &net.IPNet{IP: ip, Mask: mask}, Scope: int(netlink.SCOPE_UNIVERSE)} a := &netlink.Addr{IPNet: &net.IPNet{IP: ip, Mask: mask}, Scope: int(netlink.SCOPE_UNIVERSE)}
if err := netlink.AddrAdd(lo, a); err != nil && !errors.Is(err, os.ErrExist) { if err := netlink.AddrAdd(eth0, a); err != nil && !errors.Is(err, os.ErrExist) {
return fmt.Errorf("pod lo anycast %s: %w", ip, err) return fmt.Errorf("pod eth0 anycast %s: %w", ip, err)
} }
} }
} }