fix(image-deploy): monotonic tag guard — refuse backwards newTag writes
validate / validate (pull_request) Successful in 4s

Two concurrent release runs can race such that the older build's
image-deploy step finishes after the newer build already committed its
tag. Without an ordering check, last-writer-wins and the manifest
silently reverts to a stale image.

Add _monotonic_check: reads origin/main's current newTag for the image
and exits 0 (with a loud SKIP log) if it is numerically greater than the
tag we are trying to write. The guard runs before the kustomize edit and
again after each rebase in the push-retry loop, so both race shapes are
covered:
- Clone-then-lose: second clone already sees the newer tag → initial
  guard fires before any commit.
- Concurrent-push: both cloned the same base; one pushes first; the
  other rebases, then the post-rebase guard fires before the re-push.

Fixes bug-313bg8bgezp3
This commit is contained in:
Dave Kowalski
2026-08-24 22:59:36 +00:00
parent 082b8690b5
commit 70e1a06816
+32
View File
@@ -2,6 +2,7 @@ name: Deploy Image (kustomize image-pin in apps repo)
description: | description: |
Pin an image tag in fritzlab/apps via `kustomize edit set image`, validate the Pin an image tag in fritzlab/apps via `kustomize edit set image`, validate the
rendered manifests, and push to apps-repo main. Retries on push conflict. rendered manifests, and push to apps-repo main. Retries on push conflict.
Refuses to lower the pinned tag (monotonic write guard).
inputs: inputs:
image: image:
description: Full image name without tag (must match an entry in the target kustomization.yaml `images:` block) description: Full image name without tag (must match an entry in the target kustomization.yaml `images:` block)
@@ -52,7 +53,35 @@ runs:
git config user.name ci-bot git config user.name ci-bot
git config user.email ci-bot@fritzlab.net git config user.email ci-bot@fritzlab.net
# Read the numeric newTag currently pinned for IMAGE in origin/main.
# Returns empty string when the image entry is absent.
_origin_tag() {
git -C "$WORK" show "origin/main:${PATH_IN_REPO}/kustomization.yaml" 2>/dev/null \
| awk -v img="$IMAGE" '
/^images:/ { in_b=1 }
in_b && /- name:/ { found=(index($0, img) > 0) }
found && /newTag:/ { gsub(/[^0-9]/, ""); print; exit }
'
}
# Refuse to write a tag older than what origin/main already carries.
# Two concurrent releases can race such that the older build's deploy step
# runs after the newer build already committed its tag; without this guard
# the older run silently reverts the manifest to a stale image.
_monotonic_check() {
local existing
existing="$(_origin_tag)"
if [ -n "$existing" ] && [ "$existing" -gt "$TAG" ] 2>/dev/null; then
echo "SKIP: origin/main already has ${NAME}:${existing} > ${NAME}:${TAG} — refusing backwards write (bug-313bg8bgezp3)"
exit 0
fi
}
cd "$PATH_IN_REPO" cd "$PATH_IN_REPO"
# Monotonic guard before touching the kustomization.
_monotonic_check
kustomize edit set image "${IMAGE}=${IMAGE}:${TAG}" kustomize edit set image "${IMAGE}=${IMAGE}:${TAG}"
# Validate the kustomization renders cleanly before we push. # Validate the kustomization renders cleanly before we push.
@@ -112,6 +141,9 @@ runs:
fi fi
echo "push rejected, attempt ${ATTEMPTS}; rebasing and retrying" echo "push rejected, attempt ${ATTEMPTS}; rebasing and retrying"
git -C "$WORK" pull --rebase origin main git -C "$WORK" pull --rebase origin main
# After rebase origin/main refs are updated; re-check the monotonic
# invariant before attempting to push the rebased commit.
_monotonic_check
sleep $((ATTEMPTS * 2)) sleep $((ATTEMPTS * 2))
done done