Kubernetes limits pod.status.podIPs to one IPv4 + one IPv6 per pod.
Additional IPs in the CNI result are silently dropped by kubelet, making
the resultFromAllocation change in 4a60c00 a no-op. Revert it and add
a comment documenting the constraint so the intent is clear.
Addresses IPs remain fully functional: bound to eth0, advertised via
BGP, visible inside the pod — just not reflected in pod status.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
resultFromAllocation now appends Addresses entries to the CNI result so
they appear in pod.status.podIPs. Kubernetes and workloads that inspect
pod metadata (e.g. Plex remote-access detection) see the public IPs
alongside the IPAM-allocated ones.
Anycast IPs are intentionally excluded — they're shared across replicas
and must not appear as per-pod IPs in Kubernetes.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Like anycast, addresses IPs are advertised via BGP (/128+/32) and get
host routes via the AnycastReconciler. The sole difference: they are
assigned to pod eth0 instead of lo, so workloads that inspect their
primary interface (e.g. Plex remote-access detection) see the public IP
directly.
- annotations.go: annAddresses const, Addresses []net.IP in ParsedAnnotations
- state.go: Addresses []string persisted in allocations.json
- anycast.go: resolveAnycastTargets processes Anycast+Addresses together
- netns_linux.go: configurePodSide assigns Addresses to eth0
- netns_stub.go: mirror Addresses field for non-Linux builds
- handlers.go: thread Addresses through ADD path
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The previous implementation used JSON merge-patch (types.MergePatchType)
with a one-element conditions array. JSON merge-patch on arrays is
whole-array replacement, so every 60s flock-agent stomped over the
kubelet-managed conditions (Ready, MemoryPressure, DiskPressure,
PIDPressure), leaving only NetworkUnavailable on the node — until
kubelet's next status post (~5s later) re-set them.
Symptom: `kubectl get nodes` flickered, with one node briefly showing
Unknown each polling tick. k9s lit up red on rotating nodes. (kube-
controller-manager is also a write contender and was correctly noted
in the field-managers list.)
Switch to Server-Side Apply against the status subresource with
fieldManager=flock-agent and Force=true. NodeStatus.Conditions is a
listType=map keyed by `type`, so SSA merges by type — we declare
ownership of only the NetworkUnavailable entry and leave kubelet's
entries untouched. Force lets us reclaim the condition if a previous
CNI manager (e.g. calico-node finalizer leftovers) still owns it.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Previously the AnycastReconciler kept a pod in the nexthop set as long as
its PodReady condition was True. During a rolling restart that produces a
window after kubelet has accepted SIGTERM (DeletionTimestamp set, pod
still Ready until probes observe shutdown) where BGP still advertises a
path through the dying pod's veth — in-flight requests get RST'd when
the container actually exits.
Fix: introduce podAnycastEligible(pod) = !DeletionTimestamp && Ready,
swap it in at the AnycastReconciler's isReady callback, and fire the
ready-change callback when DeletionTimestamp transitions (the informer
UpdateFunc previously only fired on Ready transitions).
Result: as soon as the apiserver marks a pod for deletion, the
reconciler withdraws the local nexthop and BIRD reannounces the route
without it. Sibling replicas absorb traffic before the pod's
terminationGracePeriod elapses.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
K8s NetworkPolicy applies to the start of new connections; reply
packets for established flows (and ICMP related) must not be matched
against the explicit allow set. The pod ingress chain previously had
only explicit dport allows + a final drop, so any reply to a
pod-initiated outbound where the reply's dport (the ephemeral source
port) wasn't in the allow set got dropped.
Hit in production 2026-04-26: garage's `garage-admin-restrict` NP
allowed dports 3900/80/3901/3903 only. Garage uses kubernetes_discovery
to find peers — outbound to kube-apiserver succeeded, replies returned
to ephemeral source ports, dropped → "Layout not ready" cluster-wide.
Fix: emit `ct state established,related accept` as the first rule in
every pod_<hash>_(ingress|egress) chain. Regression test added.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The `pod` field hashed pod.Name, which differs per replica because of
the ReplicaSet pod-template-hash + 5-char random suffix. With
namespace,pod,image, all replicas of the same Deployment got distinct
hextets even though they were the same workload.
Replace `pod` with `app` — a stable workload identifier derived from
the controller chain:
- Deployment → ReplicaSet → Pod: strip the pod-template-hash suffix
from the RS name (`traefik-789df685f` → `traefik`).
- StatefulSet/DaemonSet/Job → Pod: use controller name as-is.
- Bare pod: pod name.
Image now comes from pod.Spec.Containers[0].Image (the spec'd
reference). 64-hex-char values are treated as sha256 digests and
parsed as before; everything else (image:tag, short SHA) is FNV-1a-64'd
as a string. This makes `traefik:v3.5` deterministic across replicas
without needing the runtime-resolved digest.
Net effect: namespace,app,image yields identical hextets across all
replicas of the same Deployment except the trailing random N nibble.
embed.Values.Pod → App; AllocRequest.Pod kept for log context only,
new App and Image fields drive the embed call. handlers.go computes
both via deriveAppName + podImageRef helpers.
Tests: 7 new TestDeriveAppName_* cases (Deploy/STS/DS/bare/RS-without-
hash/non-controller-owner) + TestPodImageRef. Existing fuzz seeds
updated for the new keyword.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add flock.fritzlab.net/ip-algo as a node-wide default via NodeConfig
metadata.annotations. Pod-level annotation still wins. Empty, missing,
or invalid input at either level falls through to the next; invalid
values warn-log via the agent's slog. Both unset → fully random IID
(unchanged baseline).
ParseAnnotations no longer touches ip-algo; ResolveIPAlgo handles the
full precedence chain, called from PodHandler.Add with the cached
NodeConfig's annotations and the agent logger.
Tests: 9 new TestResolveIPAlgo_* cases covering pod-wins, all
fall-through paths, both-absent, nil node map, whitespace, and
duplicate-as-invalid. Fuzz target rebuilt without ip-algo input space
(now exercised by ResolveIPAlgo unit tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BuiltinFamilyDefaults() now returns {WantV6: true, WantV4: true}. Pods
that want a single family explicitly opt out via the
flock.fritzlab.net/ipv4 (or ipv6) annotation, or the operator narrows
the default at the node level via NodeConfig.Spec.Defaults.
Annotation precedence is unchanged: pod annotation > NodeConfig defaults
> built-in baseline. Tests updated to reflect the new baseline; the
"opt out of v4" path now has explicit coverage.
Docs updated:
- NodeConfig.Spec.Defaults Go doc + CRD descriptions reflect the new
baseline and its overrides
- README opening framing softened from "IPv6-first" to "dual-stack,
IPv6-friendly"; example pods + spec.defaults table flipped to
treat dual-stack as the default and v6/v4-only as overrides
- README NetworkPolicy line in the comparison table flipped to
"yes (nftables)" since v1 enforcement shipped
- Limitations note about IPv4-only destinations rewritten — every
pod has v4 by default now, so the question is whether your IPv4
pool is routable beyond your network
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Move pure resolver logic out of anycast_linux.go into anycast.go so it's
unit-testable on any host. Reshape anycastTarget from a single
{hostIface, via} into a sorted list of nexthops; multiple Ready pods on
the same node binding the same anycast IP now contribute one nexthop
each.
installAnycastRoute uses RTA_MULTIPATH (via netlink.Route.MultiPath)
when the target has more than one nexthop. Single-nexthop targets keep
the simple via-route shape so 1-pod-per-node keeps rendering identically
to today's production form in `ip route show`.
flock-agent writes net.ipv{4,6}.fib_multipath_hash_policy = 1 at
startup so the kernel hashes flows on (saddr, daddr, sport, dport, proto)
rather than just IPs. Best-effort — runs privileged in production, so
it works; falls back to L3 hash on environments where the write fails
(only matters for the multi-pod-per-node case anyway).
resolveAnycastTargets sorts nexthops by canonical(via) for stable
comparison so a quiet reconcile pass doesn't churn the kernel route.
8 new unit tests cover: 1-pod, 2-pods-same-anycast (multi-nexthop),
NotReady drop, no-Ready omits the IP, pending skipped, mixed v6+v4,
family mismatch warns, determinism.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The previous base-chain jump matched iifname/oifname AND saddr/daddr ==
pod eth0 IP. Anycast traffic has the anycast IP as daddr, not the pod's
eth0 unicast — so anycast packets skipped the policy chain entirely and
fell through to the forward chain's policy=accept.
The veth uniquely belongs to one pod. Anything traversing it is to or
from that pod by definition (anycast, unicast, future overlay routes).
Match on iifname/oifname alone; let the pod-side chain's accept lines +
trailing drop be the policy.
Validated end-to-end on host001: anycast nginx pod with default-deny
ingress NetPol now correctly drops traffic from any peer; adding an
allow-from-podSelector rule unblocks only the matched peer.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
New pkg/agent/netpol implementing standard networking.k8s.io/v1
NetworkPolicy. Pipeline:
pods + policies + namespaces → Translate → Render → Apply
Supports ingress + egress, all three peer types (podSelector,
namespaceSelector, ipBlock with except), numeric ports + port ranges,
default-deny semantics derived from PolicyTypes (or inferred from
non-empty Spec.Egress when unset).
Apply path is `nft -f -` shell-out — single transaction, atomic, kernel
guarantees partial-failure rollback. Idempotent dedup via last-applied
script. Reconcile triggers: informer events, 30s self-heal tick, every
CNI ADD/DEL.
Verified against the three live cluster NetPols (calico-apiserver,
remote-proxies/lodge-home-assistant, storage/garage-admin-restrict).
Fuzz target stitches Translate + Render with random selector and peer
inputs; 21 unit tests cover the policy semantics.
Named ports skip with a warn — deferred until kubelet exposes them in a
form that doesn't require shadowing pod state.
Dockerfile: + nftables.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
NodeConfig.Spec.Defaults adds per-node IPv6/IPv4 family defaults that pod
annotations can override; built-in baseline (v6=true, v4=false) still
applies when the field is omitted.
bird.Render now validates every operator-supplied value (peer addresses,
CIDRs, anycast IPs, source addresses) before templating — fuzz found a
peer address containing `}` produced unbalanced braces in bird.conf.
Failing input preserved as a regression seed.
Fuzz targets added for ParseAnnotations, ParseCNIArgs, HostIfaceName,
canonical, IPAM allocate sequences, embed.Embed, and bird.Render.
Hardened canonical/ipToU32 against nil and non-IPv4 inputs.
README rewritten for outside readers — quickstart, NodeConfig + annotation
reference with worked examples, anycast use cases, comparison vs Calico
and Cilium, requirements, limitations.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
A single Ready/NotReady transition no longer pays a 500ms reload wait —
the first call to scheduleReload fires birdc immediately; further calls
within 500ms are coalesced into one tail reload at the cooldown's end.
Burst behavior is the same as before: under heavy churn (deploy rolling
all replicas at once), at most one reload per 500ms.
Steady-state latency from pod Ready transition to crt001 BGP withdraw:
- probe period (set in pod spec, 1s minimum)
- ~ms informer + reconcile + birdc + BGP UPDATE
The 500ms hardcoded delay is gone.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Reverts the eth0-placement hack from e1e9544. The design doc's lo
placement is correct.
Real fix: the host's anycast /128 (or /32) route now uses the pod's own
eth0 unicast IP (same family) as the route's `via` next-hop. The kernel
then does NDP/ARP for that eth0 IP — which IS configured on the pod's
eth0 — so the pod responds normally with no proxy_ndp / proxy_arp
trickery on the anycast IP itself.
ip -6 route add <anycast>/128 via <pod-eth0-v6> dev flock<8hex>
ip -4 route add <anycast>/32 via <pod-eth0-v4> dev flock<8hex>
Validation: an anycast IP whose family the pod doesn't have a unicast
for is skipped with a warn (an v4 anycast on an IPv6-only pod cannot be
NDP-resolved this way; require dual-stack).
Bonus cleanup: ESRCH from RouteDel is treated as success (idempotent).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The design doc's lo placement was motivated by avoiding NDP/ARP DAD
conflicts "across nodes advertising the same IP" — but flock pods each
sit on their own /64 veth subnet. DAD on eth0 only sees the host peer,
no cross-node L2.
With the IP on lo, the pod kernel doesn't reply to NDP solicits arriving
on eth0 (Linux default: answer NDP only for addresses on the receiving
interface). The host route `<ip>/128 dev flock<8hex>` causes the host
to do NDP for the destination on the veth; pod ignores; packet drops
silently between forwarding decision and transmit. Symptom: v4 anycast
works (proxy_arp=1 on the host veth handles ARP), v6 anycast doesn't.
Putting on eth0 makes NDP just work.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Cisco IOS rejects IPv6 BGP advertisements whose next-hop is link-local-
only. BIRD2 was synthesising a link-local next-hop for kernel-learned
routes whose dev had no via gateway (our anycast /128s). Symptom: v4
anycast worked (Cisco doesn't have the same constraint for /32s), v6
anycast didn't make it past crt001.
- pkg/routing/bird/config.go: NodeBGP.LocalV6/LocalV4. Template now
emits `local <addr> as <asn>` and `next hop self;` in the BGP
channel for both families, mirroring Calico's `source address` +
`next hop self` pattern.
- pkg/agent/bird.go: localAddrSameSubnet picks an interface address
on the peer's /64 or /24 to use as source.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
CNI ADD now adds anycast IPs to the pod's lo interface (NOT eth0 — design
doc rationale: avoid NDP/ARP DAD conflicts when N replicas share an IP).
Allocation persists the anycast list.
AnycastReconciler:
desired = { ip → flock<8hex> } from
committed allocations × pod.Status.PodReady=True
diff against advertised, install/remove host /128 (v6) or /32 (v4)
re-render bird.conf with the active set
Triggers: 2s tick, AfterCommit (per ADD/DEL), Pod informer Ready
transitions (PodCache.OnReadyChange callback).
The bird template already supported Anycast6/Anycast4 via the export
filter — this turn finally drives those slices from runtime.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
When Calico shuts down on a flock-labeled node, calico-node sets
NetworkUnavailable=True with reason CalicoIsDown. Nothing replaces it,
so kubelet's NodeController applies node.kubernetes.io/network-
unavailable:NoSchedule and new pods can't land.
flock-agent now patches Status.Conditions every 60s with
NetworkUnavailable=False (reason=FlockReady). RBAC: nodes/status patch.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Calico fenced off via Tigera Installation CR (apps@2121892). flock-agent
now renders bird.conf with the per-node BGP peers; bird sidecar reloads
on changes (debounced 500ms). Re-render tick every 15s reacts to
NodeConfig updates.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Calico's calico-node still runs on every node (Tigera-Operator-managed
via ArgoCD with selfHeal). Two birds with the same ASN can't peer to
crt001 from the same source. Use a manual static route on crt001 for
the flock /64 for the first cutover; switch to live BGP after Calico is
fenced off flock-labeled nodes.
The bird sidecar stays running with the bootstrap config (kernel +
device only, no BGP), so flipping live BGP on later is a single-line
change in runtime_linux.go.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Locks the wire format between /opt/cni/bin/flock and flock-agent. ADD
returns a CNI Result, DEL returns success/error, CHECK returns
success/error. Connection-per-RPC, newline-delimited JSON.
- pkg/cni/rpc.go: shared Op + Request + Response + framed encode/decode.
- pkg/cni/rpc_client.go: net.Dial + EncodeRequest + DecodeResponse;
rpcSocket overridable for tests.
- pkg/cni/plugin.go: real implementations of CmdAdd/Del/Check that call
through, mapping agent errors to types.Error.
- pkg/agent/rpc.go: rpcServer with swappable AddHandler/DelHandler/
CheckHandler (defaults: not-implemented for ADD; idempotent-no-op for
DEL/CHECK so kubelet teardown of a never-ADDed pod doesn't fail).
- pkg/agent/server.go: replaces the M1 accept-and-close placeholder
with rpcServer.serve(ctx, listener); listener closes on ctx cancel.
Tests cover: Request/Response JSON roundtrip, end-to-end client →
unix-socket → fake server, agent error → CNI types.Error mapping.
ADD remains "not implemented" until netlink + IPAM wire-up — the agent
returns an error and kubelet will fail pod sandbox creation IF a node
were configured to use this CNI. host001's CNI plane is still 100%
Calico, so this changes nothing observable on the cluster.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Core building block for M2 CNI ADD. Pure logic (no netlink), mutex-
serialized, seedable from committed state via MarkInUse. Hooks into
pkg/embed for ip-algo IID derivation.
- resolveEffective() implements the design-doc cidr6/cidr4 annotation
rules: equal→node, supernet→node, subnet→ann, disjoint→error.
First-match-wins across multiple annotation CIDRs.
- allocV6() random IID within the effective CIDR; on ip-algo, defers
to embed.Embed. 16-retry on collision (regenerates IID or N nibble).
- allocV4() linear scan skipping .0 (network), .1 (gateway), .<last>
(broadcast). Smallest supported block: /30 with 1 usable address.
- Deterministic fakeRand in tests covers: intersection matrix, random
IID, embed path, collision→retry, v4 skip-gateway, v4 exhaustion,
dual-stack, release-then-reallocate, family mismatch rejection.
No agent Run-loop integration yet — NewIPAM(nc.Spec.CIDR6, nc.Spec.CIDR4)
will be called from Server.Run once netlink + RPC are in place.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Agent now watches nodeconfigs.flock.fritzlab.net via a client-go dynamic
informer, filters events to its own node name, and caches the typed
NodeConfig in memory (NodeConfigCache, atomic pointer). M2's IPAM will
read from that cache.
- pkg/agent/nodeconfig.go: informer + JSON-round-trip decode (avoids
hand-written DeepCopy + scheme registration for this small a use).
- pkg/agent/server.go: starts the informer goroutine; Run terminates if
the informer returns.
- pkg/api/v1alpha1: switch placeholder TypeMeta/ObjectMeta to metav1.
- deploy/rbac: get/list/watch on nodeconfigs.
- cmd/flock-agent: --kubeconfig flag for out-of-cluster runs (tests).
Satisfies M1 verified-by: "kubectl apply NodeConfig; agent logs read it".
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Donavan Fritz