Author SHA1 Message Date
Evelyn Chen 5f4325706b fix: make CORS reconciliation recoverable
Test / contract (pull_request) Successful in 6s
2026-08-29 23:48:39 +00:00
Evelyn Chen 310ae6a29d feat(site-publish): reconcile split-surface CORS
Test / contract (pull_request) Successful in 7s
Authored-By: @architect <architect@fritzlab.net>
2026-08-29 23:41:41 +00:00
Evelyn Chen 173f0a3a6d Merge pull request '[bug-7acxk8rf0g6b] close split migration boundaries' (#4)
Test / contract (push) Successful in 7s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:25:54 +00:00
Evelyn Chen deccc4e177 test(site-publish): prove absent bucket tombstone
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:19:43 +00:00
Evelyn Chen 9b0a8c4fd4 Merge concurrent immutable history correction
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:17:33 +00:00
Evelyn Chen 898816db16 fix(site-publish): close split migration boundaries
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:15:52 +00:00
Evelyn Chen 892b6e6441 fix(site-publish): retain immutable route history
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:15:29 +00:00
Evelyn Chen 85a0b41380 fix(site-publish): close split migration boundaries
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 23:08:52 +00:00
Evelyn Chen 19fb4e43ab Merge pull request '[bug-7acxk8rf0g6b] feat(site-publish): add split-surface publishing' (#2)
Test / contract (push) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 22:35:46 +00:00
Evelyn Chen 7b824a61ca fix(site-publish): reject split input symlinks
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 22:30:43 +00:00
Evelyn Chen fb2e440bbb fix(site-publish): preflight all immutable routes
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 22:23:51 +00:00
Dave Kowalski 5261b9b99f test(site-publish): prove immutable failure ordering
Test / contract (pull_request) Successful in 7s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 22:17:24 +00:00
Evelyn Chen 5c2630b972 feat(site-publish): add split-surface publishing
Test / contract (pull_request) Successful in 6s
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
2026-08-29 22:04:50 +00:00
15 changed files with 2221 additions and 194 deletions
+14
View File
@@ -0,0 +1,14 @@
name: Test
on:
pull_request:
push:
branches: [main]
jobs:
contract:
runs-on: fritzlab
steps:
- uses: actions/checkout@v4
- name: Install test dependencies
run: python3 -m pip install --quiet --break-system-packages jinja2 pyyaml
- name: Run contract tests
run: python3 -m unittest discover -s tests -v
+163 -22
View File
@@ -1,9 +1,11 @@
# action/site-publish # action/site-publish
Composite Gitea Action that publishes a **static-content** website to the Composite Gitea Action that publishes one or more **static-content** artifacts
fritzlab k8s cluster. Supports `static`, `hugo`, and `mkdocs`. Content goes to one hostname. Each split surface owns its build input, Garage bucket,
to a Garage S3 bucket; Traefik fronts the bucket via an `ExternalName` publication credential environment variables, cache rules, Service, Ingress,
Service with cert-manager TLS. route prefix, and access middleware. The hostname shares one Certificate.
`static`, `hugo`, and `mkdocs` builds are supported; a prebuilt Docusaurus
output is a `static` artifact.
> **Containerized web apps (Dockerfile-based) are NOT handled here.** Use the > **Containerized web apps (Dockerfile-based) are NOT handled here.** Use the
> standard image-producer chain instead: > standard image-producer chain instead:
@@ -16,9 +18,10 @@ Service with cert-manager TLS.
> for the canonical example. site-publish errors out explicitly if > for the canonical example. site-publish errors out explicitly if
> `site.yaml` has `type: docker`. > `site.yaml` has `type: docker`.
## Convention ## Single-surface compatibility
Bucket name = repo name = canonical domain. Sibling hostnames (e.g. `www.`, Existing `site.yaml` files remain the `single-surface-v1` compatibility
contract. Bucket name = repo name = canonical domain. Sibling hostnames (e.g. `www.`,
`ipv6.`) are declared as `aliases:` in `site.yaml` — the action registers each `ipv6.`) are declared as `aliases:` in `site.yaml` — the action registers each
as a Garage `globalAlias` on the bucket and adds it to the Ingress + Certificate as a Garage `globalAlias` on the bucket and adds it to the Ingress + Certificate
on every deploy. Manual edits to manifests in the apps repo are clobbered; on every deploy. Manual edits to manifests in the apps repo are clobbered;
@@ -32,6 +35,144 @@ Scaffold a new site (handles repo creation + Garage bucket):
./new-site.sh --name my-site.vino.network --domain my-site.vino.network --type static ./new-site.sh --name my-site.vino.network --domain my-site.vino.network --type static
``` ```
The compatibility path still writes `build/html`, uploads with
`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, and renders `service.yaml` plus
`ingress.yaml`. Its only behavior change is the required root-cause repair:
the website Service now targets the data-only `garage-s3` Service.
## Split-surface contract
Use `artifacts` and `routes` together. This example expresses an authenticated
prebuilt portal at `/` and public bundles at `/dist`; it is illustrative and
the schema has no Baseline-specific field.
```yaml
domain: baseline.fritzlab.net
artifacts:
- name: distributions
type: static
content_dir: dist
cors_origins: ['*']
publish:
bucket: baseline-dist
credentials:
access_key_env: DIST_S3_ACCESS_KEY
secret_key_env: DIST_S3_SECRET_KEY
cache:
rules:
- path: /
cache_control: public, max-age=0, must-revalidate
- path: releases
cache_control: public, max-age=31536000, immutable
- path: channels
cache_control: public, max-age=0, must-revalidate
- name: portal
type: static
content_dir: portal/build
publish:
bucket: baseline-portal
credentials:
access_key_env: PORTAL_S3_ACCESS_KEY
secret_key_env: PORTAL_S3_SECRET_KEY
cache:
rules:
- path: /
cache_control: private, no-store
routes:
- name: distributions
path: /dist
artifact: distributions
access:
mode: public
- name: portal
path: /
artifact: portal
access:
mode: protected
middleware: authentik-forwardauth
```
The caller supplies each declared credential name as an environment variable
on the action step. Names must be matched `<NAME>_S3_ACCESS_KEY` and
`<NAME>_S3_SECRET_KEY` pairs; arbitrary environment variables cannot become
publication credentials. Values pass to `aws` only through its environment and
never appear in a logged command or process argument.
```yaml
- uses: https://code.fritzlab.net/action/site-publish@v1
with:
token: ${{ secrets.CI_BOT_TOKEN }}
env:
DIST_S3_ACCESS_KEY: ${{ secrets.DIST_S3_ACCESS_KEY }}
DIST_S3_SECRET_KEY: ${{ secrets.DIST_S3_SECRET_KEY }}
PORTAL_S3_ACCESS_KEY: ${{ secrets.PORTAL_S3_ACCESS_KEY }}
PORTAL_S3_SECRET_KEY: ${{ secrets.PORTAL_S3_SECRET_KEY }}
```
Routes are normalized and rendered longest-prefix first. Split mode requires
one `/` catch-all so unmatched paths have an explicit access policy. If any
route is protected, that catch-all must also be protected. Every artifact must
belong to exactly one route and bucket; protected and public routes cannot
reuse a bucket. A protected route requires an existing file-provider access
middleware. Public routes cannot declare one.
Every cache policy requires a `/` default. More-specific cache paths override
it, are reapplied in deterministic prefix order, and must exist in the built
artifact. Contradictory directives (`public` plus `private`, `immutable` plus
revalidation, or `no-store` plus a positive max-age) are rejected. Protected
artifacts require `private` or `no-store` and cannot emit `public`.
Metadata restamping transfers each artifact once even on a no-op publication;
that is the cost of making policy changes effective on unchanged Garage objects.
An immutable cache path is excluded from sync and deletion. Every object key in
that path must contain exactly one full publication SHA-256, calculated over its
cache policy, content type, and bytes. That content address makes concurrent
writes identical even though Garage v2.2.0 has no conditional destination
write. An identical retry converges; a changed object, missing digest metadata,
wrong address, or nested policy under that immutable prefix fails publication.
Every immutable target across every artifact is validated and published before
any route's mutable objects change.
Mutable default and override partitions receive their final cache policy before
the matching prefix-scoped stale deletion, so publication never exposes a
provisional cache policy or a pointer to a missing immutable target.
Generated Ingress annotations and `site-publish-history.yaml` retain every
seen bucket's access, prior route, and cumulative bucket-relative immutable key
prefixes, including while an artifact is absent. Removed or renamed rules stay
recorded. When a move places a retired prefix inside the new sync scope, its
immutable subtrees are excluded; a current-file collision fails publication.
The bucket-keyed history rejects a protected-to-public transition even when the
artifact is renamed; publishing that artifact publicly requires a new bucket.
Legacy single-surface is public for this downgrade check. Removing or renaming
an immutable rule preserves its prior URLs; current mutable content at one of
those paths is rejected instead of replacing it.
Artifact input directories must be pairwise disjoint after filesystem
resolution. Publication stops before build or upload if one contains another or
escapes the repository. Symlinked roots, components, and descendants are also rejected, preventing
protected input from entering a public artifact through dereference. Split
storage endpoints are pinned to Garage, and each website
authority is derived from its bucket; a site cannot expose an arbitrary backend.
`cors_origins` is reconciled as a bucket policy on every split publication. Values are either `*`
or HTTPS origins; browser access is limited to `GET` and `HEAD`. Omitting the field removes stale
CORS from that bucket. Protected artifacts cannot allow wildcard CORS. All immutable objects and
all bucket CORS policies complete before mutable channels change; if any policy write fails, the
policies already attempted are restored to their pre-publication values.
Each split route gets a bucket-specific `<bucket>.web.sjc001.fritzlab.net`
ExternalName Service annotated to disable pass-host-header and a separate Ingress. Route
Ingresses share the hostname's certificate Secret. The access middleware and
Garage bucket/key must already exist; the publisher doesn't create identity
providers or credentials.
### Migrating a site
Leave an existing single-surface file unchanged until a real second surface
exists. Then build every artifact before this action, move the old fields into
an artifact, declare a route for every artifact, give each bucket a separately
scoped key, and set the route access/cache contract. Run the repository tests
and inspect generated Apps changes. Removing a route removes its generated
Service and Ingress on the next render; bucket deletion remains manual.
Or do it manually. `site.yaml`: Or do it manually. `site.yaml`:
```yaml ```yaml
@@ -75,11 +216,11 @@ jobs:
garage-admin-token: ${{ secrets.GARAGE_ADMIN_TOKEN }} garage-admin-token: ${{ secrets.GARAGE_ADMIN_TOKEN }}
``` ```
DNS: subdomains of `vino.network` are covered by the wildcard CNAME to DNS: subdomains of `vino.network` are covered by the wildcard CNAME to the
`traefik.edge.svc…`. For other zones, add an explicit CNAME: public gateway. For other zones, add an explicit CNAME:
``` ```
my-site.fritzlab.net 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net. my-site.fritzlab.net 300 IN CNAME gateway.sjc001.fritzlab.net.
``` ```
## Inputs ## Inputs
@@ -87,10 +228,10 @@ my-site.fritzlab.net 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net.
| Input | Required | Default | Description | | Input | Required | Default | Description |
|---|---|---|---| |---|---|---|---|
| `token` | yes | | Gitea token for apps repo push | | `token` | yes | | Gitea token for apps repo push |
| `s3-access-key` | yes | | Garage `ci-deploy-key` access key id | | `s3-access-key` | legacy only | | Garage access key id for single-surface sites |
| `s3-secret-key` | yes | | Garage `ci-deploy-key` secret key | | `s3-secret-key` | legacy only | | Garage secret key for single-surface sites |
| `s3-endpoint` | no | `http://garage.storage.svc:3900` | Garage S3 endpoint | | `s3-endpoint` | no | `http://garage-s3.storage.svc:3900` | Legacy Garage S3 endpoint |
| `garage-admin-token` | only if site has `aliases` | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) | | `garage-admin-token` | legacy aliases only | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint | | `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
| `username` | no | `ci-bot` | Gitea username | | `username` | no | `ci-bot` | Gitea username |
@@ -107,12 +248,11 @@ Org secrets in `websites`: `CI_BOT_TOKEN`, `GARAGE_S3_ACCESS_KEY`,
``` ```
push to websites/<repo> push to websites/<repo>
→ CI runs site-publish action → CI runs site-publish action
→ reads site.yaml, builds content (static copy / hugo / mkdocs), runs tidy → reads and validates all of site.yaml before publication
aws s3 sync → Garage bucket named after the repo independently builds each static / Hugo / MkDocs artifact
admin API: ensures every alias from site.yaml is a globalAlias on the bucket syncs each artifact to its route-owned Garage bucket and prefix
→ renders manifests in fritzlab/apps from templates: ExternalName Service → → reapplies the artifact's default and longest-prefix cache headers
garage.storage.svc, Traefik Ingress (canonical + aliases), cert-manager → renders one Service + Ingress per route and one shared Certificate
Certificate (canonical + aliases as SANs), kustomization
→ commits + pushes apps repo only if diff is non-empty → commits + pushes apps repo only if diff is non-empty
→ ArgoCD syncs → site live with TLS → ArgoCD syncs → site live with TLS
``` ```
@@ -120,9 +260,10 @@ push to websites/<repo>
The Ingress + Certificate are re-rendered on every deploy from `site.yaml`. The Ingress + Certificate are re-rendered on every deploy from `site.yaml`.
There is no "first-deploy vs. update" branching — every deploy is idempotent. There is no "first-deploy vs. update" branching — every deploy is idempotent.
No nginx pods, no per-site Docker images. Garage matches `Host:` header to No nginx pods, no per-site Docker images. Compatibility sites pass the public
bucket name (or any of its globalAliases), so every site shares a single host to the shared data-only Garage website Service. Split routes disable host
ExternalName target. passing on their Service so Garage receives that artifact's bucket-specific
website authority.
## History ## History
+6 -6
View File
@@ -1,15 +1,15 @@
name: Publish Site name: Publish Site
description: Build and deploy a static-content site (static, hugo, mkdocs) to Garage S3 with Traefik + cert-manager. Containerized apps should use action/image-build + action/image-push + action/image-deploy. description: Build and deploy one or more routed static-content artifacts to Garage S3 with Traefik and cert-manager.
inputs: inputs:
token: token:
description: Gitea token (ci-bot) for apps repo push and API operations description: Gitea token (ci-bot) for apps repo push and API operations
required: true required: true
s3-access-key: s3-access-key:
description: Garage ci-deploy-key access key id description: Garage access key id (required by the legacy single-surface contract)
required: true required: false
s3-secret-key: s3-secret-key:
description: Garage ci-deploy-key secret access key description: Garage secret access key (required by the legacy single-surface contract)
required: true required: false
s3-endpoint: s3-endpoint:
# Targets garage-s3 (data-only Service) so requests do not round-robin onto # Targets garage-s3 (data-only Service) so requests do not round-robin onto
# the gateway pod, whose emptyDir-backed metadata view intermittently # the gateway pod, whose emptyDir-backed metadata view intermittently
@@ -18,7 +18,7 @@ inputs:
required: false required: false
default: http://garage-s3.storage.svc:3900 default: http://garage-s3.storage.svc:3900
garage-admin-token: garage-admin-token:
description: Garage admin API token (required only when site.yaml has aliases — used to reconcile bucket globalAliases) description: Garage admin API token (required only for legacy aliases — used to reconcile bucket globalAliases)
required: false required: false
garage-admin-endpoint: garage-admin-endpoint:
description: Garage admin API endpoint URL description: Garage admin API endpoint URL
+2 -2
View File
@@ -165,6 +165,6 @@ echo
echo "Site created: ${ORG}/${NAME}" echo "Site created: ${ORG}/${NAME}"
echo "First build will trigger on push." echo "First build will trigger on push."
echo echo
echo "DNS: ${DOMAIN} is covered by the *.vino.network wildcard (→ traefik.edge)." echo "DNS: ${DOMAIN} is covered by the *.vino.network wildcard (→ public gateway)."
echo "For a domain outside vino.network, add an explicit CNAME:" echo "For a domain outside vino.network, add an explicit CNAME:"
echo " ${DOMAIN} 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net." echo " ${DOMAIN} 300 IN CNAME gateway.sjc001.fritzlab.net."
+26 -21
View File
@@ -1,25 +1,25 @@
"""Build phase — content prep for static-content sites.""" """Build each declared static-content artifact independently."""
import shutil import shutil
import subprocess import subprocess
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from utils import EXCLUDE_FILES, env, parse_site_yaml, run from utils import EXCLUDE_FILES, env, parse_site_yaml, run, validate_artifact_inputs
def build_static(site_dir, cfg): def build_artifact(site_dir, artifact):
build_dir = site_dir / "build" html_dir = site_dir / artifact["build_dir"]
html_dir = build_dir / "html" if html_dir.parent.exists():
shutil.rmtree(html_dir.parent)
if build_dir.exists(): content_dir = artifact["content_dir"]
shutil.rmtree(build_dir)
content_dir = cfg["content_dir"]
src = site_dir / content_dir if content_dir else site_dir src = site_dir / content_dir if content_dir else site_dir
if not src.exists():
raise FileNotFoundError(f"artifact {artifact['name']} content_dir not found: {src}")
if cfg["type"] == "static": if artifact["type"] == "static":
print(f"Copying static content from {src}") print(f"Copying artifact {artifact['name']} from {src}")
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp) / "html" tmp_path = Path(tmp) / "html"
shutil.copytree(src, tmp_path, dirs_exist_ok=True) shutil.copytree(src, tmp_path, dirs_exist_ok=True)
@@ -29,18 +29,18 @@ def build_static(site_dir, cfg):
shutil.rmtree(p) shutil.rmtree(p)
elif p.exists(): elif p.exists():
p.unlink() p.unlink()
build_dir.mkdir(parents=True) html_dir.parent.mkdir(parents=True)
shutil.move(str(tmp_path), str(html_dir)) shutil.move(str(tmp_path), str(html_dir))
elif cfg["type"] == "hugo": elif artifact["type"] == "hugo":
print(f"Building Hugo site from {src}") print(f"Building Hugo artifact {artifact['name']} from {src}")
run(f"hugo --source {src} --destination {html_dir}") run(["hugo", "--source", str(src), "--destination", str(html_dir)])
elif cfg["type"] == "mkdocs": elif artifact["type"] == "mkdocs":
print(f"Building MkDocs site from {src}") print(f"Building MkDocs artifact {artifact['name']} from {src}")
run(f"cd {src} && mkdocs build -d {html_dir}") run(["mkdocs", "build", "-d", str(html_dir)], cwd=src)
if cfg.get("tidy", True): if artifact["tidy"]:
print("Running tidy on HTML files...") print("Running tidy on HTML files...")
for html_file in html_dir.rglob("*.html"): for html_file in html_dir.rglob("*.html"):
subprocess.run( subprocess.run(
@@ -51,7 +51,9 @@ def build_static(site_dir, cfg):
check=False, check=False,
) )
print(f"Build complete — content at {html_dir}") if not any(path.is_file() for path in html_dir.rglob("*")):
raise FileNotFoundError(f"artifact {artifact['name']} produced no files in {html_dir}")
print(f"Artifact {artifact['name']} complete — content at {html_dir}")
def cmd_build(): def cmd_build():
@@ -62,4 +64,7 @@ def cmd_build():
print("Site disabled — skipping build") print("Site disabled — skipping build")
return return
build_static(site_dir, cfg) validate_artifact_inputs(site_dir, cfg)
for artifact in cfg["artifacts"]:
build_artifact(site_dir, artifact)
+486 -61
View File
@@ -1,17 +1,21 @@
"""Deploy phase — S3 sync, manifest rendering, alias reconcile.""" """Deploy phase — S3 sync, manifest rendering, alias reconcile."""
import fnmatch
import hashlib
import json import json
import mimetypes
import os import os
import shlex import re
import shutil import shutil
import subprocess
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from urllib.error import HTTPError, URLError from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen from urllib.request import Request, urlopen
import yaml
from utils import ( from utils import (
DEFAULT_S3_ENDPOINT,
GITEA_HOST,
NAMESPACE, NAMESPACE,
clone_apps, clone_apps,
commit_and_push, commit_and_push,
@@ -21,6 +25,7 @@ from utils import (
parse_site_yaml, parse_site_yaml,
render_templates, render_templates,
run, run,
validate_artifact_inputs,
) )
GARAGE_ADMIN_ENDPOINT = os.environ.get( GARAGE_ADMIN_ENDPOINT = os.environ.get(
@@ -28,44 +33,337 @@ GARAGE_ADMIN_ENDPOINT = os.environ.get(
) )
CACHE_CONTROL = "public, max-age=0, must-revalidate" def validate_artifact_output(site_dir, artifact):
"""Prove every artifact and declared cache prefix exists before publishing any."""
html_dir = site_dir / artifact["build_dir"]
if not html_dir.is_dir() or not any(path.is_file() for path in html_dir.rglob("*")):
die(f"artifact {artifact['name']} build output is absent or empty: {html_dir}")
for rule in artifact["cache_rules"]:
if not rule["path"]:
continue
cache_root = html_dir / rule["path"]
if not cache_root.exists() or not any(path.is_file() for path in cache_root.rglob("*")):
die(f"artifact {artifact['name']} cache path /{rule['path']} has no built files")
def s3_sync(site_name, site_dir, excludes=None): def validate_publication_environment(cfg):
endpoint = os.environ.get("GARAGE_S3_ENDPOINT", DEFAULT_S3_ENDPOINT) """Resolve every declared credential before the first bucket is changed."""
html_dir = site_dir / "build" / "html" for artifact in cfg["artifacts"]:
if not html_dir.exists(): env(artifact["credentials"]["access_key_env"])
die(f"build/html not found — did the build step run? ({html_dir})") env(artifact["credentials"]["secret_key_env"])
env("AWS_ACCESS_KEY_ID") if cfg["compatibility"] and cfg["aliases"] and not os.environ.get("GARAGE_ADMIN_TOKEN"):
env("AWS_SECRET_ACCESS_KEY") die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
os.environ.setdefault("AWS_DEFAULT_REGION", "sjc001")
def _is_immutable(rule):
return "immutable" in {
part.strip().lower().split("=", 1)[0]
for part in rule["cache_control"].split(",")
}
def _aws_capture(args, aws_env):
"""Run a non-streaming AWS request without exposing environment credentials."""
print(f" $ {' '.join(str(part) for part in args)}")
return subprocess.run(args, env=aws_env, text=True, capture_output=True, check=False)
def _immutable_head(endpoint, bucket, key, aws_env):
args = ["aws", "--endpoint-url", endpoint, "s3api", "head-object",
"--bucket", bucket, "--key", key, "--output", "json"]
result = _aws_capture(args, aws_env)
if result.returncode == 0:
return json.loads(result.stdout)
error = f"{result.stdout}\n{result.stderr}"
if any(marker in error for marker in ("404", "Not Found", "NoSuchKey")):
return None
raise RuntimeError(f"head-object failed for s3://{bucket}/{key}: {error.strip()}")
def _immutable_digests(source, cache_control, content_type):
with source.open("rb") as stream:
content_digest = hashlib.file_digest(stream, "sha256").hexdigest()
publication = hashlib.sha256()
publication.update(cache_control.encode())
publication.update(b"\0")
publication.update(content_type.encode())
publication.update(b"\0")
with source.open("rb") as stream:
for block in iter(lambda: stream.read(1024 * 1024), b""):
publication.update(block)
return content_digest, publication.hexdigest()
def _same_immutable_object(info, content_digest, publication_digest, cache_control, content_type):
metadata = {key.lower(): value for key, value in (info.get("Metadata") or {}).items()}
return (
metadata.get("sha256") == content_digest
and metadata.get("publication-sha256") == publication_digest
and info.get("CacheControl") == cache_control
and info.get("ContentType") == content_type
)
def publish_immutable_file(endpoint, bucket, key, source, cache_control, aws_env):
"""Publish a content-addressed key; identical retries converge."""
content_type = mimetypes.guess_type(source.name)[0] or "application/octet-stream"
content_digest, publication_digest = _immutable_digests(source, cache_control, content_type)
address_digests = re.findall(r"(?<![0-9a-f])([0-9a-f]{64})(?![0-9a-f])", key.lower())
if address_digests != [publication_digest]:
raise RuntimeError(
f"immutable key must contain its one publication SHA-256 {publication_digest}: "
f"s3://{bucket}/{key}"
)
existing = _immutable_head(endpoint, bucket, key, aws_env)
if existing is not None:
if _same_immutable_object(
existing, content_digest, publication_digest, cache_control, content_type,
):
print(f" Immutable object already matches: s3://{bucket}/{key}")
return False
raise RuntimeError(f"immutable object differs or lacks publisher digest: s3://{bucket}/{key}")
args = ["aws", "--endpoint-url", endpoint, "s3api", "put-object",
"--bucket", bucket, "--key", key, "--body", str(source),
"--content-type", content_type, "--cache-control", cache_control,
"--metadata", f"sha256={content_digest},publication-sha256={publication_digest}"]
result = _aws_capture(args, aws_env)
if result.returncode == 0:
return True
error = f"{result.stdout}\n{result.stderr}"
raise RuntimeError(f"put-object failed for s3://{bucket}/{key}: {error.strip()}")
def publish_immutable_rule(artifact, route, rule, html_dir, aws_env):
"""Publish one immutable cache partition without overwrite or deletion."""
rule_root = html_dir / rule["path"]
child_paths = [candidate["path"] for candidate in artifact["cache_rules"]
if candidate["path"].startswith(f"{rule['path'].rstrip('/')}/")]
object_prefix = route["path"].strip("/")
for source in sorted(path for path in rule_root.rglob("*") if path.is_file()):
relative = source.relative_to(html_dir).as_posix()
if any(relative == child or relative.startswith(f"{child}/") for child in child_paths):
continue
if any(fnmatch.fnmatch(relative, pattern) for pattern in artifact["excludes"]):
continue
key = "/".join(part for part in (object_prefix, relative) if part)
publish_immutable_file(
artifact["s3_endpoint"], artifact["bucket"], key, source,
rule["cache_control"], aws_env,
)
def publication_aws_env(artifact, credential_env_names=None):
"""Build the route-scoped AWS environment without leaking other credentials."""
access_key = env(artifact["credentials"]["access_key_env"])
secret_key = env(artifact["credentials"]["secret_key_env"])
aws_env = os.environ.copy()
for name in credential_env_names or artifact["credentials"].values():
aws_env.pop(name, None)
for name in ("CI_BOT_TOKEN", "GARAGE_ADMIN_TOKEN", "AWS_PROFILE",
"AWS_SHARED_CREDENTIALS_FILE", "AWS_SESSION_TOKEN"):
aws_env.pop(name, None)
aws_env.update({
"AWS_ACCESS_KEY_ID": access_key,
"AWS_SECRET_ACCESS_KEY": secret_key,
"AWS_DEFAULT_REGION": os.environ.get("AWS_DEFAULT_REGION", "sjc001"),
})
return aws_env
def configure_cors(bucket, origins, endpoint, aws_env):
"""Reconcile read-only browser access without exposing publication credentials."""
if origins is None:
return
config = None
if origins:
config = {
"CORSRules": [{
"AllowedOrigins": origins,
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 3600,
}],
}
set_cors_configuration(bucket, config, endpoint, aws_env)
def set_cors_configuration(bucket, config, endpoint, aws_env):
"""Apply an exact bucket CORS configuration, or remove it when absent."""
if config is None:
run([
"aws", "--endpoint-url", endpoint, "s3api", "delete-bucket-cors",
"--bucket", bucket,
], env=aws_env)
return
with tempfile.NamedTemporaryFile("w", suffix=".json", encoding="utf-8") as handle:
json.dump(config, handle)
handle.flush()
run([
"aws", "--endpoint-url", endpoint, "s3api", "put-bucket-cors",
"--bucket", bucket, "--cors-configuration", f"file://{handle.name}",
], env=aws_env)
def get_cors_configuration(bucket, endpoint, aws_env):
"""Read the exact bucket CORS configuration for rollback."""
result = _aws_capture([
"aws", "--endpoint-url", endpoint, "s3api", "get-bucket-cors",
"--bucket", bucket, "--output", "json",
], aws_env)
if result.returncode == 0:
try:
config = json.loads(result.stdout)
except json.JSONDecodeError as error:
raise RuntimeError(f"get-bucket-cors returned invalid JSON for {bucket}") from error
if not isinstance(config, dict) or not isinstance(config.get("CORSRules"), list):
raise RuntimeError(f"get-bucket-cors returned an invalid policy for {bucket}")
return config
error = f"{result.stdout}\n{result.stderr}"
if "NoSuchCORSConfiguration" in error:
return None
raise RuntimeError(f"get-bucket-cors failed for {bucket}: {error.strip()}")
def reconcile_artifact_cors(artifacts, credential_env_names=None):
"""Reconcile all policies, restoring the prior set if any write fails."""
snapshots = []
for artifact in artifacts:
if artifact["cors_origins"] is None:
continue
aws_env = publication_aws_env(artifact, credential_env_names)
snapshots.append((
artifact,
aws_env,
get_cors_configuration(artifact["bucket"], artifact["s3_endpoint"], aws_env),
))
attempted = []
try:
for artifact, aws_env, previous in snapshots:
attempted.append((artifact, aws_env, previous))
configure_cors(
artifact["bucket"], artifact["cors_origins"], artifact["s3_endpoint"], aws_env,
)
except Exception as error:
rollback_errors = []
for artifact, aws_env, previous in reversed(attempted):
try:
set_cors_configuration(
artifact["bucket"], previous, artifact["s3_endpoint"], aws_env,
)
except Exception as rollback_error:
rollback_errors.append(f"{artifact['bucket']}: {rollback_error}")
if rollback_errors:
raise RuntimeError(
f"CORS reconciliation failed ({error}); rollback also failed for "
f"{'; '.join(rollback_errors)}"
) from error
raise
def publish_route_immutables(artifact, route, site_dir, credential_env_names=None):
"""Publish one route's immutable partitions during the global preflight."""
html_dir = site_dir / artifact["build_dir"]
aws_env = publication_aws_env(artifact, credential_env_names)
for rule in artifact["cache_rules"]:
if _is_immutable(rule):
publish_immutable_rule(artifact, route, rule, html_dir, aws_env)
def immutable_key_prefixes(artifact, route):
"""Return immutable partitions as bucket-relative key prefixes."""
route_prefix = route["path"].strip("/")
return [
"/".join(part for part in (route_prefix, rule["path"]) if part)
for rule in artifact["cache_rules"] if _is_immutable(rule)
]
def retained_immutable_paths(artifact, route, previous_contract):
"""Carry all bucket history forward so later route moves cannot delete it."""
previous_paths = previous_contract["immutable_paths"] if previous_contract else []
return sorted(set(previous_paths) | set(immutable_key_prefixes(artifact, route)))
def retired_immutable_filters(artifact, route, html_dir, previous_contract):
"""Protect historical immutable keys that fall inside the current sync scope."""
if not previous_contract:
return []
current_prefix = route["path"].strip("/")
filters = []
current_immutable = set(immutable_key_prefixes(artifact, route))
for immutable_path in previous_contract["immutable_paths"]:
if immutable_path in current_immutable:
continue
if current_prefix:
marker = f"{current_prefix}/"
if not immutable_path.startswith(marker):
continue
retired_path = immutable_path[len(marker):]
else:
retired_path = immutable_path
collision = html_dir / retired_path
if collision.exists() and any(path.is_file() for path in collision.rglob("*")):
raise RuntimeError(
f"current artifact collides with retired immutable partition: {retired_path}"
)
filters.extend(("--exclude", f"{retired_path}/*"))
return filters
def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_contract=None):
endpoint = artifact["s3_endpoint"]
html_dir = site_dir / artifact["build_dir"]
aws_env = publication_aws_env(artifact, credential_env_names)
bucket = artifact["bucket"]
object_prefix = route["path"].strip("/")
destination = f"s3://{bucket}/{object_prefix + '/' if object_prefix else ''}"
default_cache = next(rule["cache_control"] for rule in artifact["cache_rules"] if not rule["path"])
immutable_paths = [rule["path"] for rule in artifact["cache_rules"] if _is_immutable(rule)]
# `excludes` are patterns (site.yaml `excludes:` list) that should never # `excludes` are patterns (site.yaml `excludes:` list) that should never
# be uploaded *and* should never be deleted from the bucket — escape hatch # be uploaded *and* should never be deleted from the bucket — escape hatch
# for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli). # for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli).
exclude_flags = " ".join(f"--exclude {shlex.quote(p)}" for p in (excludes or [])) exclude_args = [arg for pattern in artifact["excludes"] for arg in ("--exclude", pattern)]
if excludes: if artifact["excludes"]:
print(f"Excluding patterns: {excludes}") print(f"Excluding patterns: {artifact['excludes']}")
print(f"Syncing {html_dir}s3://{site_name} via {endpoint}") print(f"Syncing artifact {artifact['name']}{destination} via {endpoint}")
# `sync --delete` handles new/changed/orphaned files. `cp --recursive` # Upload with the final cache policy before cleanup. Sync and deletion are
# then re-uploads everything to refresh metadata (cache-control, # scoped to the same current route prefix and cache partition. A route move
# content-type) on objects sync skipped because nothing changed. # leaves its old bucket partition intact but unreachable after the old
# Cost: a no-op deploy still re-uploads every byte. Sites here are # Ingress disappears, while stale mutable keys on the serving prefix are
# small enough that that's free; correctness wins over throughput. # deleted. Immutable subtrees are structurally excluded. `cp --recursive`
# refreshes metadata atomically per object before `sync --delete` removes
# stale keys without ever exposing new bytes under a provisional policy.
# A no-op deploy therefore transfers the artifact bytes once.
# AWS CLI guesses Content-Type from file extension on local→S3 uploads, # AWS CLI guesses Content-Type from file extension on local→S3 uploads,
# so a fresh upload always carries the right MIME type. # so a fresh upload always carries the right MIME type.
run( specific_paths = [rule["path"] for rule in artifact["cache_rules"] if rule["path"]]
f"aws --endpoint-url {endpoint} s3 sync {html_dir}/ s3://{site_name}/ " default_filters = [arg for path in specific_paths for arg in ("--exclude", f"{path}/*")]
f"--delete --only-show-errors " retired_filters = retired_immutable_filters(artifact, route, html_dir, previous_contract)
f"--cache-control '{CACHE_CONTROL}' " run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
f"{exclude_flags}".rstrip() "--recursive", "--only-show-errors", "--cache-control", default_cache,
) *default_filters, *retired_filters, *exclude_args], env=aws_env)
print("Re-stamping metadata on all objects...") run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
run( "--delete", "--only-show-errors", "--cache-control", default_cache,
f"aws --endpoint-url {endpoint} s3 cp {html_dir}/ s3://{site_name}/ " *default_filters, *retired_filters, *exclude_args], env=aws_env)
f"--recursive --only-show-errors " for rule in artifact["cache_rules"]:
f"--cache-control '{CACHE_CONTROL}' " if not rule["path"]:
f"{exclude_flags}".rstrip() continue
) if _is_immutable(rule):
continue
include = f"{rule['path'].rstrip('/')}/*"
child_filters = [arg for path in specific_paths
if path.startswith(f"{rule['path'].rstrip('/')}/")
for arg in ("--exclude", f"{path}/*")]
# Apply rules from the artifact root so artifact-level exclusions keep
# their original meaning under every cache override.
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
"--recursive", "--only-show-errors", "--cache-control", rule["cache_control"],
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
"--delete", "--only-show-errors", "--cache-control", rule["cache_control"],
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
def garage_admin(method, path, token, body=None): def garage_admin(method, path, token, body=None):
@@ -89,15 +387,13 @@ def ensure_bucket_aliases(site_name, aliases, admin_token):
if not aliases: if not aliases:
return return
if not admin_token: if not admin_token:
print(" (no GARAGE_ADMIN_TOKEN — skipping bucket alias reconcile)") die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
return
try: try:
info = garage_admin("GET", f"/v2/GetBucketInfo?globalAlias={site_name}", info = garage_admin("GET", f"/v2/GetBucketInfo?globalAlias={site_name}",
admin_token) admin_token)
except (HTTPError, URLError) as e: except (HTTPError, URLError) as e:
print(f" WARNING: bucket lookup failed: {e}") raise RuntimeError(f"bucket lookup failed for {site_name}: {e}") from e
return
bucket_id = info.get("id") bucket_id = info.get("id")
existing = set(info.get("globalAliases") or []) existing = set(info.get("globalAliases") or [])
@@ -116,50 +412,178 @@ def ensure_bucket_aliases(site_name, aliases, admin_token):
raise raise
def render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg): def render_site_manifests(
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts=None,
):
"""Always re-render manifests from current site.yaml. Templates own """Always re-render manifests from current site.yaml. Templates own
domain + aliases, so changes propagate without manual edits.""" domain + aliases, so changes propagate without manual edits."""
manifests_dir.mkdir(parents=True, exist_ok=True) manifests_dir.mkdir(parents=True, exist_ok=True)
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
routes = []
previous_contracts = previous_contracts or {}
next_contracts = {bucket: dict(contract)
for bucket, contract in previous_contracts.items()}
for route in cfg["routes"]:
artifact = artifact_by_name[route["artifact"]]
resource_name = k8s_name(site_name) if cfg["compatibility"] else f"{k8s_name(site_name)}-{route['name']}"
previous = previous_contracts.get(artifact["bucket"])
immutable_paths = retained_immutable_paths(artifact, route, previous)
next_contracts[artifact["bucket"]] = {
"path": route["path"],
"access": "public" if route["access"] == "legacy" else route["access"],
"artifact": route["artifact"],
"immutable_paths": immutable_paths,
}
routes.append({
**route, "resource_name": resource_name, "artifact_config": artifact,
"immutable_paths_json": json.dumps(immutable_paths, separators=(",", ":")),
})
template_vars = { template_vars = {
"site": site_name, "site": site_name,
"site_k8s": k8s_name(site_name), "site_k8s": k8s_name(site_name),
"domain": cfg["domain"], "domain": cfg["domain"],
"aliases": cfg["aliases"], "aliases": cfg["aliases"],
"namespace": NAMESPACE, "namespace": NAMESPACE,
"middlewares": cfg["middlewares"], "compatibility": cfg["compatibility"],
"routes": routes,
} }
render_templates(action_dir, template_vars, app_dir, manifests_dir) render_templates(action_dir, template_vars, app_dir, manifests_dir)
if not cfg["compatibility"] or previous_contracts:
(app_dir / "site-publish-history.yaml").write_text(yaml.safe_dump(
{"version": 1, "buckets": next_contracts}, sort_keys=True,
))
def _validate_route_contract(bucket, contract, path):
expected = {"path", "access", "artifact", "immutable_paths"}
if (not isinstance(bucket, str) or not bucket or not isinstance(contract, dict)
or set(contract) != expected
or contract.get("access") not in {"public", "protected"}
or not isinstance(contract.get("path"), str)
or not contract["path"].startswith("/")
or not isinstance(contract.get("artifact"), str) or not contract["artifact"]
or not isinstance(contract.get("immutable_paths"), list)
or any(not isinstance(item, str) or not item or item.startswith("/")
or any(part in {"", ".", ".."} for part in item.split("/"))
for item in contract["immutable_paths"])):
raise RuntimeError(f"invalid site-publish route history in {path}")
return {
"path": contract["path"], "access": contract["access"],
"artifact": contract["artifact"],
"immutable_paths": sorted(set(contract["immutable_paths"])),
}
def previous_route_contracts(app_dir):
"""Read bucket-keyed route history from generated Ingresses."""
history_path = app_dir / "site-publish-history.yaml"
if history_path.exists():
document = yaml.safe_load(history_path.read_text())
if (not isinstance(document, dict) or set(document) != {"version", "buckets"}
or document["version"] != 1 or not isinstance(document["buckets"], dict)):
raise RuntimeError(f"invalid site-publish route history in {history_path}")
return {
bucket: _validate_route_contract(bucket, contract, history_path)
for bucket, contract in document["buckets"].items()
}
contracts = {}
manifests = app_dir / "manifests"
if not manifests.exists():
return contracts
for path in sorted(manifests.glob("ingress*.yaml")):
document = yaml.safe_load(path.read_text()) or {}
annotations = document.get("metadata", {}).get("annotations", {})
artifact = annotations.get("site-publish.fritzlab.net/artifact")
access = annotations.get("site-publish.fritzlab.net/access")
bucket = annotations.get("site-publish.fritzlab.net/bucket")
immutable_paths_json = annotations.get("site-publish.fritzlab.net/immutable-paths")
route_path = annotations.get("site-publish.fritzlab.net/route-path")
values = (artifact, access, bucket, immutable_paths_json, route_path)
if all(value is None for value in values):
continue
if (not all(isinstance(value, str) for value in values)
or access not in {"public", "protected"} or not route_path.startswith("/")):
raise RuntimeError(f"invalid site-publish route history in {path}")
try:
immutable_paths = json.loads(immutable_paths_json)
except json.JSONDecodeError as exc:
raise RuntimeError(f"invalid site-publish route history in {path}") from exc
if not isinstance(immutable_paths, list) or any(not isinstance(item, str) for item in immutable_paths):
raise RuntimeError(f"invalid site-publish route history in {path}")
if bucket in contracts:
raise RuntimeError(f"duplicate site-publish route history for bucket {bucket}")
contracts[bucket] = _validate_route_contract(bucket, {
"path": route_path, "access": access, "artifact": artifact,
"immutable_paths": immutable_paths,
}, path)
return contracts
def validate_route_migrations(cfg, previous_contracts):
artifacts = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
for route in cfg["routes"]:
artifact = artifacts[route["artifact"]]
previous = previous_contracts.get(artifact["bucket"])
if (previous and previous["access"] == "protected"
and route["access"] in {"public", "legacy"}
):
raise RuntimeError(
f"artifact {route['artifact']} cannot become public while reusing protected "
f"bucket {artifact['bucket']}"
)
def deploy_static(site_name, site_dir, action_dir, token, cfg): def deploy_static(site_name, site_dir, action_dir, token, cfg):
s3_sync(site_name, site_dir, excludes=cfg.get("excludes")) artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN")) credential_env_names = {
name for artifact in cfg["artifacts"] for name in artifact["credentials"].values()
}
validate_publication_environment(cfg)
for artifact in cfg["artifacts"]:
validate_artifact_output(site_dir, artifact)
apps_dir = clone_apps(token) apps_dir = clone_apps(token)
app_dir = apps_dir / "sjc001" / "websites" / site_name app_dir = apps_dir / "sjc001" / "websites" / site_name
manifests_dir = app_dir / "manifests" manifests_dir = app_dir / "manifests"
previous_contracts = previous_route_contracts(app_dir)
validate_route_migrations(cfg, previous_contracts)
# Complete immutable work across the whole publication before any route's
# mutable pointers can change. Partial immutable success is safe; mixing a
# new route with an old route after a later immutable failure is not.
for route in cfg["routes"]:
publish_route_immutables(
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
)
# Reconcile every browser-read policy before publishing mutable content.
# A CORS failure therefore cannot leave a new channel pointing at a release
# whose cross-origin assets browsers cannot consume.
reconcile_artifact_cors(cfg["artifacts"], credential_env_names)
for route in cfg["routes"]:
s3_sync(
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
previous_contracts.get(artifact_by_name[route["artifact"]]["bucket"]),
)
if cfg["compatibility"]:
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg) render_site_manifests(
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts,
)
commit_and_push(apps_dir, f"Deploy {site_name}") commit_and_push(apps_dir, f"Deploy {site_name}", token)
def decommission(site_name, token): def decommission(site_name, token, buckets=None):
"""Remove manifests from apps repo.""" """Remove manifests from apps repo."""
user = env("CI_BOT_USER", "ci-bot") apps_dir = clone_apps(token)
with tempfile.TemporaryDirectory() as tmp: site_path = apps_dir / "sjc001" / "websites" / site_name
apps_dir = Path(tmp) if not site_path.exists():
run(f"git clone --depth 1 https://{user}:{token}@{GITEA_HOST}/fritzlab/apps.git {apps_dir}") print(f"No manifests for {site_name} — nothing to remove")
site_path = apps_dir / "sjc001" / "websites" / site_name return
if not site_path.exists(): shutil.rmtree(site_path)
print(f"No manifests for {site_name} — nothing to remove") commit_and_push(apps_dir, f"Decommission {site_name}", token)
return for bucket in buckets or [site_name]:
shutil.rmtree(site_path) print(f"Bucket {bucket} and its objects are NOT purged automatically.")
run(f"git -C {apps_dir} config user.name {user}") print(f" garage bucket delete {bucket} --yes")
run(f"git -C {apps_dir} config user.email {user}@fritzlab.net")
commit_and_push(apps_dir, f"Decommission {site_name}")
print(f"Bucket {site_name} and its objects are NOT purged automatically.")
print(f" garage bucket delete {site_name} --yes")
def cmd_deploy(): def cmd_deploy():
@@ -173,7 +597,8 @@ def cmd_deploy():
if not cfg["enabled"]: if not cfg["enabled"]:
print("Site disabled — running decommission...") print("Site disabled — running decommission...")
decommission(site_name, token) decommission(site_name, token, [artifact["bucket"] for artifact in cfg["artifacts"]])
return return
validate_artifact_inputs(site_dir, cfg)
deploy_static(site_name, site_dir, action_dir, token, cfg) deploy_static(site_name, site_dir, action_dir, token, cfg)
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
case "$1" in
*Username*) printf '%s\n' "$CI_BOT_USER" ;;
*) printf '%s\n' "$CI_BOT_TOKEN" ;;
esac
+3 -2
View File
@@ -24,9 +24,10 @@ def ensure_aws():
subprocess.run(["aws", "--version"], check=True) subprocess.run(["aws", "--version"], check=True)
def ensure_jinja2(): def ensure_python_dependencies():
try: try:
import jinja2 import jinja2
import yaml
except ImportError: except ImportError:
print("Installing jinja2 + pyyaml...") print("Installing jinja2 + pyyaml...")
subprocess.run( subprocess.run(
@@ -36,6 +37,6 @@ def ensure_jinja2():
if __name__ == "__main__": if __name__ == "__main__":
ensure_jinja2() ensure_python_dependencies()
ensure_aws() ensure_aws()
print("Setup complete") print("Setup complete")
+525 -69
View File
@@ -1,26 +1,35 @@
"""Shared utilities for the site-publish action.""" """Shared utilities for the site-publish action."""
import ipaddress
import os import os
import re
import shutil import shutil
import subprocess import subprocess
import sys import sys
from pathlib import Path from pathlib import Path, PurePosixPath
from urllib.parse import urlparse
import yaml import yaml
from jinja2 import Environment, FileSystemLoader from jinja2 import Environment, FileSystemLoader, StrictUndefined
APPS_REPO = "fritzlab/apps" APPS_REPO = "fritzlab/apps"
GITEA_HOST = "code.fritzlab.net" GITEA_HOST = "code.fritzlab.net"
NAMESPACE = "websites" NAMESPACE = "websites"
DEFAULT_S3_ENDPOINT = "http://garage-s3.storage.svc:3900" DEFAULT_S3_ENDPOINT = "http://garage-s3.storage.svc:3900"
DEFAULT_WEBSITE_SUFFIX = "web.sjc001.fritzlab.net"
DEFAULT_CACHE_CONTROL = "public, max-age=0, must-revalidate"
EXCLUDE_FILES = { EXCLUDE_FILES = {
".git", ".gitea", ".gitignore", "site.yaml", ".git", ".gitea", ".gitignore", "site.yaml", "build", ".site-publish",
"build", "Makefile", "README.md", "CLAUDE.md", "Makefile", "README.md", "CLAUDE.md", "Dockerfile", ".dockerignore",
"Dockerfile", ".dockerignore", "go.mod", "go.sum", "go.mod", "go.sum",
} }
VALID_TYPES = {"static", "hugo", "mkdocs"} VALID_TYPES = {"static", "hugo", "mkdocs"}
NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")
ENV_RE = re.compile(r"^[A-Z_][A-Z0-9_]*$")
ACCESS_KEY_ENV_RE = re.compile(r"^([A-Z][A-Z0-9_]*)_S3_ACCESS_KEY$")
BUCKET_RE = re.compile(r"^[a-z0-9](?:[a-z0-9.-]{1,61}[a-z0-9])?$")
MIDDLEWARE_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9])?$")
DOCKER_DEPRECATION_MSG = """\ DOCKER_DEPRECATION_MSG = """\
type: docker is no longer supported by action/site-publish. type: docker is no longer supported by action/site-publish.
@@ -42,113 +51,560 @@ example.\
""" """
class ConfigError(ValueError):
"""A site.yaml contract violation."""
def k8s_name(name): def k8s_name(name):
"""Sanitize for DNS-1035 label (dots → dashes)."""
return name.replace(".", "-") return name.replace(".", "-")
def env(key, default=None): def env(key, default=None):
val = os.environ.get(key, default) val = os.environ.get(key, default)
if val is None: if val is None or val == "":
die(f"Missing required env var: {key}") die(f"Missing required env var: {key}")
return val return val
def die(msg): def die(msg):
print(f"ERROR: {msg}", file=sys.stderr) print(f"ERROR: {msg}", file=sys.stderr)
sys.exit(1) raise SystemExit(1)
def run(cmd, **kwargs): def run(cmd, *, display=None, **kwargs):
print(f" $ {cmd}") """Run an argv command, printing only a safe display form."""
return subprocess.run(cmd, shell=True, check=True, **kwargs) if not isinstance(cmd, (list, tuple)):
raise TypeError("run() requires an argv list")
shown = display if display is not None else " ".join(str(part) for part in cmd)
print(f" $ {shown}")
return subprocess.run(cmd, check=True, **kwargs)
def parse_site_yaml(site_dir): def _mapping(value, label):
path = Path(site_dir) / "site.yaml" if not isinstance(value, dict):
if not path.exists(): raise ConfigError(f"{label} must be a mapping")
die("site.yaml not found in repo root") return value
with open(path) as f:
cfg = yaml.safe_load(f)
if not cfg.get("domain"): def _list(value, label):
die("domain is required in site.yaml") if not isinstance(value, list):
raise ConfigError(f"{label} must be a list")
return value
site_type = cfg.get("type", "static")
def _known_keys(value, allowed, label):
unknown = set(value) - set(allowed)
if unknown:
raise ConfigError(f"{label} has unknown fields: {', '.join(sorted(unknown))}")
def _strings(value, label):
values = _list(value or [], label)
if any(not isinstance(item, str) or not item for item in values):
raise ConfigError(f"{label} must be a list of non-empty strings")
return values
def _cors_origins(value, label):
origins = _list(value, label)
if any(not isinstance(item, str) or not item for item in origins):
raise ConfigError(f"{label} must be a list of non-empty strings")
canonical = []
for origin in origins:
if origin == "*":
canonical.append(origin)
continue
parsed = urlparse(origin)
try:
port = parsed.port
except ValueError:
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins") from None
if parsed.scheme != "https" or not parsed.hostname or parsed.path or parsed.params or (
parsed.query or parsed.fragment or parsed.username or parsed.password
):
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
try:
address = ipaddress.ip_address(parsed.hostname)
except ValueError:
try:
hostname = parsed.hostname.encode("idna").decode("ascii")
except UnicodeError:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
) from None
try:
_hostname(hostname, f"{label} hostname")
except ConfigError:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
) from None
else:
if getattr(address, "scope_id", None) is not None:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
)
hostname = f"[{address.compressed}]" if address.version == 6 else address.compressed
canonical.append(f"https://{hostname}{f':{port}' if port not in (None, 443) else ''}")
if len(canonical) != len(set(canonical)):
raise ConfigError(f"{label} must not contain duplicate canonical origins")
if "*" in canonical and len(canonical) != 1:
raise ConfigError(f"{label} wildcard must be the only origin")
if origins != canonical:
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
return origins
def _hostname(value, label):
if not isinstance(value, str) or len(value) > 253 or value.endswith("."):
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
labels = value.split(".")
if len(labels) < 2 or any(not NAME_RE.fullmatch(part) for part in labels):
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
return value
def _aliases(value, domain):
aliases = _strings(value or [], "aliases")
aliases = [_hostname(alias, "aliases entry") for alias in aliases]
if len(aliases) != len(set(aliases)) or domain in aliases:
raise ConfigError("aliases must be unique and cannot repeat domain")
return aliases
def _relative_path(value, label, *, allow_root=False):
value = "" if value is None else value
if not isinstance(value, str):
raise ConfigError(f"{label} must be a string")
if value == "/" and allow_root:
return ""
path = PurePosixPath(value)
if path.is_absolute() or ".." in path.parts:
raise ConfigError(f"{label} must be a relative path without '..'")
normalized = str(path).strip("/")
return "" if normalized == "." else normalized
def _route_path(value, label):
if not isinstance(value, str) or not value.startswith("/"):
raise ConfigError(f"{label} must start with '/'")
if "//" in value or "?" in value or "#" in value or ".." in value.split("/"):
raise ConfigError(f"{label} is not a canonical URL path")
normalized = value.rstrip("/") or "/"
if not re.fullmatch(r"/[A-Za-z0-9._~/-]*", normalized):
raise ConfigError(f"{label} contains unsupported URL path characters")
return normalized
def _middlewares(value, label):
names = _strings(value, label)
if any(not MIDDLEWARE_RE.fullmatch(name) for name in names):
raise ConfigError(f"{label} contains an invalid middleware name")
if len(names) != len(set(names)):
raise ConfigError(f"{label} contains duplicate middleware names")
return names
def _cache_control(value, label):
if not isinstance(value, str) or not value.strip():
raise ConfigError(f"{label} must be a non-empty Cache-Control value")
directives = [part.strip().lower() for part in value.split(",")]
names = [part.split("=", 1)[0] for part in directives]
if len(names) != len(set(names)):
raise ConfigError(f"{label} repeats a Cache-Control directive")
present = set(names)
if {"public", "private"} <= present:
raise ConfigError(f"{label} cannot be both public and private")
ages = {}
for part in directives:
name = part.split("=", 1)[0]
if name in {"max-age", "s-maxage"}:
raw = part.split("=", 1)[1] if "=" in part else ""
if not raw.isdigit():
raise ConfigError(f"{label} {name} must be a non-negative integer")
ages[name] = int(raw)
max_age = ages.get("max-age")
if "immutable" in present and (not max_age or present & {"no-store", "no-cache", "must-revalidate"}):
raise ConfigError(f"{label} immutable requires positive max-age without revalidation")
if "no-store" in present and any(ages.values()):
raise ConfigError(f"{label} no-store contradicts positive caching")
if "private" in present and ages.get("s-maxage"):
raise ConfigError(f"{label} private contradicts shared-cache max-age")
return ", ".join(part.strip() for part in value.split(","))
def _site_type(value, label):
site_type = value or "static"
if site_type == "docker": if site_type == "docker":
die(DOCKER_DEPRECATION_MSG) raise ConfigError(DOCKER_DEPRECATION_MSG)
if site_type not in VALID_TYPES: if site_type not in VALID_TYPES:
die(f"Unknown site type: {site_type} (valid: {', '.join(sorted(VALID_TYPES))})") raise ConfigError(f"Unknown {label}: {site_type} (valid: {', '.join(sorted(VALID_TYPES))})")
return site_type
excludes = cfg.get("excludes") or []
if not isinstance(excludes, list) or any(not isinstance(p, str) for p in excludes):
die("excludes must be a list of string patterns")
middlewares = cfg.get("middlewares") or [] def _endpoint(value, label):
if not isinstance(middlewares, list) or any(not isinstance(m, str) for m in middlewares): parsed = urlparse(value)
die("middlewares must be a list of Traefik file-provider middleware names") if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.path not in {"", "/"}:
raise ConfigError(f"{label} must be an http(s) origin without a path")
return value.rstrip("/")
site = {
"domain": cfg["domain"], def _legacy_config(raw, site_name):
"type": site_type, if not isinstance(raw.get("tidy", True), bool):
"enabled": cfg.get("enabled", True), raise ConfigError("tidy must be a boolean")
"aliases": cfg.get("aliases") or [], if not isinstance(raw.get("enabled", True), bool):
"content_dir": cfg.get("content_dir", ""), raise ConfigError("enabled must be a boolean")
"tidy": cfg.get("tidy", True), artifact = {
"excludes": excludes, "name": "site",
"type": _site_type(raw.get("type", "static"), "site type"),
"content_dir": _relative_path(raw.get("content_dir", ""), "content_dir"),
"tidy": raw.get("tidy", True),
"excludes": _strings(raw.get("excludes") or [], "excludes"),
"build_dir": "build/html",
"bucket": site_name,
"s3_endpoint": os.environ.get("GARAGE_S3_ENDPOINT") or DEFAULT_S3_ENDPOINT,
"website_authority": "garage-s3.storage.svc.k8s.sjc001.fritzlab.net",
"credentials": {"access_key_env": "AWS_ACCESS_KEY_ID", "secret_key_env": "AWS_SECRET_ACCESS_KEY"},
"cache_rules": [{"path": "", "cache_control": DEFAULT_CACHE_CONTROL}],
"cors_origins": None,
}
return {
"version": 1,
"compatibility": "single-surface-v1",
"domain": raw["domain"],
"aliases": _aliases(raw.get("aliases"), raw["domain"]),
"enabled": raw.get("enabled", True),
"artifacts": [artifact],
"routes": [{
"name": "site", "path": "/", "artifact": "site", "access": "legacy",
"access_middleware": None,
"middlewares": _middlewares(raw.get("middlewares") or [], "middlewares"),
}],
}
def _artifact(item, index):
label = f"artifacts[{index}]"
item = _mapping(item, label)
_known_keys(
item,
{"name", "type", "content_dir", "tidy", "excludes", "publish", "cache", "cors_origins"},
label,
)
name = item.get("name")
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
raise ConfigError(f"{label}.name must be a DNS label")
publish = _mapping(item.get("publish"), f"{label}.publish")
_known_keys(publish, {"bucket", "credentials"}, f"{label}.publish")
bucket = publish.get("bucket")
if not isinstance(bucket, str) or not BUCKET_RE.fullmatch(bucket):
raise ConfigError(f"{label}.publish.bucket is not a valid bucket name")
credentials = _mapping(publish.get("credentials"), f"{label}.publish.credentials")
_known_keys(credentials, {"access_key_env", "secret_key_env"}, f"{label}.publish.credentials")
normalized_credentials = {}
for key in ("access_key_env", "secret_key_env"):
value = credentials.get(key)
if not isinstance(value, str) or not ENV_RE.fullmatch(value):
raise ConfigError(f"{label}.publish.credentials.{key} must name an environment variable")
normalized_credentials[key] = value
access_match = ACCESS_KEY_ENV_RE.fullmatch(normalized_credentials["access_key_env"])
expected_secret = (
f"{access_match.group(1)}_S3_SECRET_KEY" if access_match else None
)
if normalized_credentials["secret_key_env"] != expected_secret:
raise ConfigError(
f"{label}.publish.credentials must be a matched "
"<NAME>_S3_ACCESS_KEY and <NAME>_S3_SECRET_KEY pair"
)
cache = _mapping(item.get("cache"), f"{label}.cache")
_known_keys(cache, {"rules"}, f"{label}.cache")
rules = _list(cache.get("rules"), f"{label}.cache.rules")
if not rules:
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
cache_rules, paths = [], set()
for rule_index, rule in enumerate(rules):
rule_label = f"{label}.cache.rules[{rule_index}]"
rule = _mapping(rule, rule_label)
_known_keys(rule, {"path", "cache_control"}, rule_label)
path = _relative_path(rule.get("path"), f"{rule_label}.path", allow_root=True)
if path in paths:
raise ConfigError(f"{label}.cache.rules has duplicate path /{path}")
paths.add(path)
cache_rules.append({"path": path, "cache_control": _cache_control(
rule.get("cache_control"), f"{rule_label}.cache_control"
)})
if "" not in paths:
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
cache_rules.sort(key=lambda rule: (len(PurePosixPath(rule["path"]).parts), rule["path"]))
immutable_paths = [rule["path"] for rule in cache_rules
if "immutable" in {part.strip().lower().split("=", 1)[0]
for part in rule["cache_control"].split(",")}]
if "" in immutable_paths:
raise ConfigError(f"{label}.cache.rules immutable paths must be narrower than '/'")
for path in immutable_paths:
if any(other.startswith(f"{path}/") for other in paths):
raise ConfigError(f"{label}.cache.rules cannot nest another policy under immutable /{path}")
authority = f"{bucket}.{DEFAULT_WEBSITE_SUFFIX}"
if not isinstance(item.get("tidy", True), bool):
raise ConfigError(f"{label}.tidy must be a boolean")
return {
"name": name,
"type": _site_type(item.get("type", "static"), f"{label}.type"),
"content_dir": _relative_path(item.get("content_dir", ""), f"{label}.content_dir"),
"tidy": item.get("tidy", True),
"excludes": _strings(item.get("excludes") or [], f"{label}.excludes"),
"build_dir": f".site-publish/{name}/html",
"bucket": bucket,
"s3_endpoint": DEFAULT_S3_ENDPOINT,
"website_authority": authority,
"credentials": normalized_credentials,
"cache_rules": cache_rules,
"cors_origins": (
_cors_origins(item["cors_origins"], f"{label}.cors_origins")
if "cors_origins" in item else []
),
}
def _route(item, index):
label = f"routes[{index}]"
item = _mapping(item, label)
_known_keys(item, {"name", "path", "artifact", "access", "middlewares"}, label)
name = item.get("name")
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
raise ConfigError(f"{label}.name must be a DNS label")
access = _mapping(item.get("access"), f"{label}.access")
_known_keys(access, {"mode", "middleware"}, f"{label}.access")
mode, middleware = access.get("mode"), access.get("middleware")
if mode not in {"public", "protected"}:
raise ConfigError(f"{label}.access.mode must be public or protected")
if mode == "protected" and (not isinstance(middleware, str) or not MIDDLEWARE_RE.fullmatch(middleware)):
raise ConfigError(f"{label}.access.middleware is required for protected access")
if mode == "public" and middleware is not None:
raise ConfigError(f"{label}.access.middleware is forbidden for public access")
middlewares = _middlewares(item.get("middlewares") or [], f"{label}.middlewares")
if middleware in middlewares:
raise ConfigError(f"{label} repeats its access middleware")
artifact = item.get("artifact")
if not isinstance(artifact, str):
raise ConfigError(f"{label}.artifact must name an artifact")
return {
"name": name, "path": _route_path(item.get("path"), f"{label}.path"),
"artifact": artifact, "access": mode, "access_middleware": middleware,
"middlewares": middlewares, "middlewares": middlewares,
} }
def _validate_multi(cfg):
artifacts, routes = cfg["artifacts"], cfg["routes"]
artifact_names = [item["name"] for item in artifacts]
route_names = [item["name"] for item in routes]
route_paths = [item["path"] for item in routes]
if len(artifact_names) != len(set(artifact_names)):
raise ConfigError("artifact names must be unique")
if len(route_names) != len(set(route_names)):
raise ConfigError("route names must be unique")
if len(route_paths) != len(set(route_paths)):
raise ConfigError("route paths are ambiguous after normalization")
if "/" not in route_paths:
raise ConfigError("routes must declare a '/' catch-all")
artifact_by_name = {item["name"]: item for item in artifacts}
references = {name: [] for name in artifact_by_name}
for route in routes:
if route["artifact"] not in artifact_by_name:
raise ConfigError(f"route {route['name']} references unknown artifact {route['artifact']}")
references[route["artifact"]].append(route)
for name, used_by in references.items():
if len(used_by) != 1:
raise ConfigError(f"artifact {name} must be referenced by exactly one route")
buckets, authorities, credential_owners = {}, {}, {}
for route in routes:
artifact = artifact_by_name[route["artifact"]]
if artifact["bucket"] in buckets:
other_access, other_name = buckets[artifact["bucket"]]
if other_access != route["access"]:
raise ConfigError(f"bucket {artifact['bucket']} cannot be reused by protected and public routes")
raise ConfigError(f"bucket {artifact['bucket']} must belong to one artifact ({other_name})")
buckets[artifact["bucket"]] = (route["access"], artifact["name"])
if artifact["website_authority"] in authorities:
raise ConfigError(
f"website authority {artifact['website_authority']} must belong to one artifact"
)
authorities[artifact["website_authority"]] = artifact["name"]
for variable in artifact["credentials"].values():
if variable in credential_owners:
raise ConfigError(
f"publication credential {variable} is reused by artifacts "
f"{credential_owners[variable]} and {artifact['name']}"
)
credential_owners[variable] = artifact["name"]
for cache_rule in artifact["cache_rules"]:
directives = {part.strip().lower().split("=", 1)[0]
for part in cache_rule["cache_control"].split(",")}
if route["access"] == "protected" and "public" in directives:
raise ConfigError(f"protected route {route['name']} cannot use public cache policy")
if route["access"] == "protected" and not directives & {"private", "no-store"}:
raise ConfigError(f"protected route {route['name']} cache policy must be private or no-store")
if route["access"] == "protected" and "s-maxage" in directives:
raise ConfigError(f"protected route {route['name']} cannot use shared-cache max-age")
if route["access"] == "public" and "private" in directives:
raise ConfigError(f"public route {route['name']} cannot use private cache policy")
if route["access"] == "protected" and "*" in artifact["cors_origins"]:
raise ConfigError(
f"protected route {route['name']} cannot allow wildcard CORS"
)
root = next(route for route in routes if route["path"] == "/")
if any(route["access"] == "protected" for route in routes) and root["access"] == "public":
raise ConfigError("a public '/' catch-all would expose unmatched protected content")
def normalize_site_config(raw, site_name):
raw = _mapping(raw, "site.yaml")
domain = _hostname(raw.get("domain"), "domain")
if not isinstance(raw.get("enabled", True), bool):
raise ConfigError("enabled must be a boolean")
if ("artifacts" in raw) != ("routes" in raw):
raise ConfigError("artifacts and routes must be declared together")
if "artifacts" not in raw:
return _legacy_config(raw, site_name)
legacy_fields = {"type", "content_dir", "tidy", "excludes", "middlewares"} & set(raw)
if legacy_fields:
raise ConfigError("legacy fields cannot be mixed with artifacts/routes: " + ", ".join(sorted(legacy_fields)))
_known_keys(raw, {"domain", "aliases", "enabled", "artifacts", "routes"}, "site.yaml")
artifacts = [_artifact(item, index) for index, item in enumerate(_list(raw["artifacts"], "artifacts"))]
routes = [_route(item, index) for index, item in enumerate(_list(raw["routes"], "routes"))]
if not artifacts or not routes:
raise ConfigError("artifacts and routes must not be empty")
cfg = {
"version": 2, "compatibility": None, "domain": domain,
"aliases": _aliases(raw.get("aliases"), domain),
"enabled": raw.get("enabled", True),
"artifacts": sorted(artifacts, key=lambda item: item["name"]),
"routes": sorted(routes, key=lambda item: (-len(item["path"]), item["path"], item["name"])),
}
_validate_multi(cfg)
for route in cfg["routes"]:
if len(f"{k8s_name(site_name)}-{route['name']}") > 63:
raise ConfigError(f"route {route['name']} makes the generated Service name exceed 63 characters")
return cfg
def validate_artifact_inputs(site_dir, cfg):
"""Reject source containment before a public or protected build starts."""
if cfg["compatibility"]:
return
root = Path(site_dir).resolve()
sources = []
for artifact in cfg["artifacts"]:
declared = root
for component in Path(artifact["content_dir"]).parts:
declared /= component
if declared.is_symlink():
raise ConfigError(
f"artifact {artifact['name']} content_dir contains symlink component: "
f"{declared.relative_to(root)}"
)
source = declared.resolve()
if source != root and root not in source.parents:
raise ConfigError(
f"artifact {artifact['name']} content_dir resolves outside the repository"
)
if source.exists():
symlink = next((path for path in source.rglob("*") if path.is_symlink()), None)
if symlink is not None:
raise ConfigError(
f"artifact {artifact['name']} build input contains symlink: "
f"{symlink.relative_to(root)}"
)
sources.append((artifact["name"], source))
for index, (name, source) in enumerate(sources):
for other_name, other_source in sources[index + 1:]:
if source == other_source or source in other_source.parents or other_source in source.parents:
raise ConfigError(
f"artifact build inputs overlap after resolution: {name} and {other_name}"
)
def parse_site_yaml(site_dir, site_name=None):
path = Path(site_dir) / "site.yaml"
if not path.exists():
die("site.yaml not found in repo root")
if site_name is None:
repo = os.environ.get("SITE_REPO", "")
site_name = repo.split("/", 1)[-1] if "/" in repo else Path(site_dir).name
try:
with open(path) as stream:
cfg = normalize_site_config(yaml.safe_load(stream), site_name)
except (ConfigError, yaml.YAMLError) as exc:
die(str(exc))
print("Site config:") print("Site config:")
for k, v in site.items(): print(f" domain: {cfg['domain']}")
print(f" {k}: {v}") print(f" contract: {cfg['compatibility'] or 'split-surface-v2'}")
return site print(f" artifacts: {[item['name'] for item in cfg['artifacts']]}")
print(f" routes: {[(item['path'], item['access']) for item in cfg['routes']]}")
return cfg
def clone_apps(token): def clone_apps(token):
"""Clone Apps without placing the credential in argv or output."""
user = env("CI_BOT_USER", "ci-bot") user = env("CI_BOT_USER", "ci-bot")
apps_dir = Path("/tmp/apps-deploy") apps_dir = Path("/tmp/apps-deploy")
if apps_dir.exists(): if apps_dir.exists():
shutil.rmtree(apps_dir) shutil.rmtree(apps_dir)
run(f"git clone --depth 1 https://{user}:{token}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}") clone_env = git_auth_env(token)
run(f"git -C {apps_dir} config user.name {user}") url = f"https://{user}@{GITEA_HOST}/{APPS_REPO}.git"
run(f"git -C {apps_dir} config user.email {user}@fritzlab.net") run(["git", "clone", "--depth", "1", url, str(apps_dir)],
display=f"git clone --depth 1 https://{user}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}", env=clone_env)
run(["git", "-C", str(apps_dir), "config", "user.name", user])
run(["git", "-C", str(apps_dir), "config", "user.email", f"{user}@fritzlab.net"])
return apps_dir return apps_dir
def render_templates(action_dir, template_vars, app_dir, manifests_dir): def render_templates(action_dir, template_vars, app_dir, manifests_dir):
"""Render Jinja2 templates for a static-content site.""" """Render one certificate and deterministic per-route resources."""
templates_dir = Path(action_dir) / "templates" jinja_env = Environment(loader=FileSystemLoader(str(Path(action_dir) / "templates")),
jinja_env = Environment( keep_trailing_newline=True, undefined=StrictUndefined)
loader=FileSystemLoader(str(templates_dir)), manifests_dir.mkdir(parents=True, exist_ok=True)
keep_trailing_newline=True, for child in manifests_dir.iterdir():
) if child.is_file() and child.suffix in {".yaml", ".yml"}:
child.unlink()
tmpl_names = ["app.yaml.j2", "certificate.yaml.j2", "ingress.yaml.j2", route_files = []
"kustomization.yaml.j2", "service.yaml.j2"] for route in template_vars["routes"]:
stem = "" if template_vars["compatibility"] else f"-{route['name']}"
for tmpl_name in tmpl_names: for kind in ("service", "ingress"):
tmpl = jinja_env.get_template(tmpl_name) out_name = f"{kind}{stem}.yaml"
rendered = tmpl.render(**template_vars) destination = manifests_dir / out_name
out_name = tmpl_name.replace(".j2", "") destination.write_text(jinja_env.get_template(f"{kind}.yaml.j2").render(
dest = app_dir / out_name if tmpl_name == "app.yaml.j2" else manifests_dir / out_name **template_vars, route=route
dest.write_text(rendered) ))
print(f" Rendered {tmpl_name} -> {dest}") route_files.append(out_name)
print(f" Rendered {kind}.yaml.j2 -> {destination}")
common_vars = {**template_vars, "route_files": route_files}
for out_name, destination in {
"certificate.yaml": manifests_dir / "certificate.yaml",
"kustomization.yaml": manifests_dir / "kustomization.yaml",
"app.yaml": app_dir / "app.yaml",
}.items():
destination.write_text(jinja_env.get_template(f"{out_name}.j2").render(**common_vars))
print(f" Rendered {out_name}.j2 -> {destination}")
def commit_and_push(apps_dir, message): def git_auth_env(token):
run(f"git -C {apps_dir} add -A") """Return credential-safe Git authentication shared by clone and push."""
result = subprocess.run( auth_env = os.environ.copy()
f"git -C {apps_dir} diff --cached --quiet", auth_env["CI_BOT_TOKEN"] = token
shell=True, check=False, auth_env["GIT_ASKPASS"] = str(Path(__file__).with_name("git-askpass.sh"))
) auth_env["GIT_TERMINAL_PROMPT"] = "0"
return auth_env
def commit_and_push(apps_dir, message, token=None):
run(["git", "-C", str(apps_dir), "add", "-A"])
result = subprocess.run(["git", "-C", str(apps_dir), "diff", "--cached", "--quiet"], check=False)
if result.returncode == 0: if result.returncode == 0:
print("No manifest changes to commit") print("No manifest changes to commit")
return False return False
run(f"git -C {apps_dir} commit -m '{message}'") run(["git", "-C", str(apps_dir), "commit", "-m", message])
run(f"git -C {apps_dir} push") push_env = git_auth_env(token) if token else None
run(["git", "-C", str(apps_dir), "push"], env=push_env)
print("Manifests pushed — ArgoCD will sync") print("Manifests pushed — ArgoCD will sync")
return True return True
+12 -7
View File
@@ -1,12 +1,17 @@
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
kind: Ingress kind: Ingress
metadata: metadata:
name: {{ site_k8s }} name: {{ route.resource_name }}
namespace: {{ namespace }} namespace: {{ namespace }}
{%- if site_type != "docker" %}
annotations: annotations:
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% for m in middlewares %},{{ m }}@file{% endfor %} {%- if not compatibility %}
site-publish.fritzlab.net/artifact: {{ route.artifact | tojson }}
site-publish.fritzlab.net/access: {{ route.access | tojson }}
site-publish.fritzlab.net/bucket: {{ route.artifact_config.bucket | tojson }}
site-publish.fritzlab.net/immutable-paths: {{ route.immutable_paths_json | tojson }}
site-publish.fritzlab.net/route-path: {{ route.path | tojson }}
{%- endif %} {%- endif %}
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% if route.access_middleware %},{{ route.access_middleware }}@file{% endif %}{% for m in route.middlewares %},{{ m }}@file{% endfor %}
spec: spec:
ingressClassName: traefik ingressClassName: traefik
tls: tls:
@@ -20,22 +25,22 @@ spec:
- host: {{ domain }} - host: {{ domain }}
http: http:
paths: paths:
- path: / - path: {{ route.path }}
pathType: Prefix pathType: Prefix
backend: backend:
service: service:
name: {{ site_k8s }} name: {{ route.resource_name }}
port: port:
number: 80 number: 80
{%- for alias in aliases %} {%- for alias in aliases %}
- host: {{ alias }} - host: {{ alias }}
http: http:
paths: paths:
- path: / - path: {{ route.path }}
pathType: Prefix pathType: Prefix
backend: backend:
service: service:
name: {{ site_k8s }} name: {{ route.resource_name }}
port: port:
number: 80 number: 80
{%- endfor %} {%- endfor %}
+3 -2
View File
@@ -1,6 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1 apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization kind: Kustomization
resources: resources:
- service.yaml {% for route_file in route_files -%}
- ingress.yaml - {{ route_file }}
{% endfor -%}
- certificate.yaml - certificate.yaml
+6 -2
View File
@@ -1,11 +1,15 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ site_k8s }} name: {{ route.resource_name }}
namespace: {{ namespace }} namespace: {{ namespace }}
{%- if not compatibility %}
annotations:
traefik.ingress.kubernetes.io/service.passhostheader: "false"
{%- endif %}
spec: spec:
type: ExternalName type: ExternalName
externalName: garage.storage.svc.k8s.sjc001.fritzlab.net externalName: {{ route.artifact_config.website_authority }}
ports: ports:
- port: 80 - port: 80
targetPort: 80 targetPort: 80
+9
View File
@@ -0,0 +1,9 @@
domain: example.fritzlab.net
type: static
content_dir: html
aliases:
- www.example.fritzlab.net
middlewares:
- response-headers
excludes:
- media/*
+43
View File
@@ -0,0 +1,43 @@
domain: baseline.fritzlab.net
artifacts:
- name: portal
type: static
content_dir: portal/build
publish:
bucket: baseline-portal
credentials:
access_key_env: PORTAL_S3_ACCESS_KEY
secret_key_env: PORTAL_S3_SECRET_KEY
cache:
rules:
- path: /
cache_control: private, no-store
- name: distributions
type: static
content_dir: dist
cors_origins: ['*']
publish:
bucket: baseline-dist
credentials:
access_key_env: DIST_S3_ACCESS_KEY
secret_key_env: DIST_S3_SECRET_KEY
cache:
rules:
- path: /
cache_control: public, max-age=0, must-revalidate
- path: releases
cache_control: public, max-age=31536000, immutable
- path: channels
cache_control: public, max-age=0, must-revalidate
routes:
- name: portal
path: /
artifact: portal
access:
mode: protected
middleware: authentik-forwardauth
- name: distributions
path: /dist
artifact: distributions
access:
mode: public
+918
View File
@@ -0,0 +1,918 @@
import base64
import copy
import hashlib
import io
import json
import os
import socket
import subprocess
import sys
import tempfile
import threading
import unittest
from contextlib import redirect_stderr, redirect_stdout
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
from unittest.mock import patch
import yaml
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "scripts"))
import deploy
import build
import utils
from utils import ConfigError, normalize_site_config, validate_artifact_inputs
def fixture(name):
return yaml.safe_load((ROOT / "tests" / "fixtures" / name).read_text())
class IPv6GitHTTPServer(ThreadingHTTPServer):
address_family = socket.AF_INET6
class AuthenticatedGitHandler(BaseHTTPRequestHandler):
project_root = None
expected_authorization = None
def log_message(self, _format, *_args):
pass
def do_GET(self):
self._git_backend()
def do_POST(self):
self._git_backend()
def _git_backend(self):
if self.headers.get("Authorization") != self.expected_authorization:
self.send_response(401)
self.send_header("WWW-Authenticate", 'Basic realm="test"')
self.end_headers()
return
parsed = urlsplit(self.path)
length = int(self.headers.get("Content-Length", "0"))
request_body = self.rfile.read(length) if length else b""
backend_env = os.environ.copy()
backend_env.update({
"GIT_PROJECT_ROOT": str(self.project_root),
"GIT_HTTP_EXPORT_ALL": "1",
"PATH_INFO": parsed.path,
"QUERY_STRING": parsed.query,
"REQUEST_METHOD": self.command,
"CONTENT_TYPE": self.headers.get("Content-Type", ""),
"CONTENT_LENGTH": str(length),
"REMOTE_USER": "ci-bot",
"REMOTE_ADDR": "::1",
"GATEWAY_INTERFACE": "CGI/1.1",
"SERVER_PROTOCOL": "HTTP/1.1",
})
result = subprocess.run(
["git", "http-backend"], input=request_body, env=backend_env,
capture_output=True, check=True,
)
raw_headers, response_body = result.stdout.split(b"\r\n\r\n", 1)
headers, status = [], 200
for line in raw_headers.decode().split("\r\n"):
name, value = line.split(":", 1)
if name.lower() == "status":
status = int(value.strip().split(" ", 1)[0])
else:
headers.append((name, value.strip()))
self.send_response(status)
for name, value in headers:
self.send_header(name, value)
self.end_headers()
self.wfile.write(response_body)
class ConfigContractTests(unittest.TestCase):
def setUp(self):
self.raw = fixture("split-site.yaml")
def assert_invalid(self, mutate, message):
raw = copy.deepcopy(self.raw)
mutate(raw)
with self.assertRaisesRegex(ConfigError, message):
normalize_site_config(raw, "baseline.fritzlab.net")
def test_legacy_normalizes_to_explicit_compatibility_surface(self):
cfg = normalize_site_config(fixture("legacy-site.yaml"), "example.fritzlab.net")
self.assertEqual("single-surface-v1", cfg["compatibility"])
self.assertEqual("build/html", cfg["artifacts"][0]["build_dir"])
self.assertEqual("example.fritzlab.net", cfg["artifacts"][0]["bucket"])
self.assertEqual("/", cfg["routes"][0]["path"])
self.assertEqual("legacy", cfg["routes"][0]["access"])
def test_routes_are_sorted_longest_prefix_first(self):
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
self.assertEqual(["/dist", "/"], [route["path"] for route in cfg["routes"]])
credentials = {artifact["name"]: artifact["credentials"] for artifact in cfg["artifacts"]}
self.assertNotEqual(credentials["distributions"], credentials["portal"])
distributions = next(
artifact for artifact in cfg["artifacts"] if artifact["name"] == "distributions"
)
self.assertEqual(["*"], distributions["cors_origins"])
def test_cors_origins_are_https_origins_or_wildcard(self):
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["http://consumer.example"]
),
"must contain '\\*' or canonical HTTPS origins",
)
for origin in (
"https://consumer example",
"https://Consumer.example",
"https://consumer.example:443",
"https://consumer.example/",
"https://consumer.example:invalid",
"https://[fe80::1%eth0]",
"https://[fe80::1%25eth0]",
):
with self.subTest(origin=origin):
self.assert_invalid(
lambda raw, origin=origin: raw["artifacts"][1].__setitem__(
"cors_origins", [origin]
),
"must contain '\\*' or canonical HTTPS origins",
)
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["https://consumer.example", "https://consumer.example:443"]
),
"must not contain duplicate canonical origins",
)
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["*", "https://consumer.example"]
),
"wildcard must be the only origin",
)
raw = copy.deepcopy(self.raw)
raw["artifacts"][1]["cors_origins"] = ["https://[2602:817:3000::1]:8443"]
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
self.assertEqual(
["https://[2602:817:3000::1]:8443"],
next(a for a in cfg["artifacts"] if a["name"] == "distributions")["cors_origins"],
)
def test_declared_cors_origins_must_be_a_list(self):
for value in (None, False, 0, "", {}):
with self.subTest(value=value):
self.assert_invalid(
lambda raw, value=value: raw["artifacts"][1].__setitem__(
"cors_origins", value
),
"cors_origins must be a list",
)
def test_protected_route_rejects_wildcard_cors(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0].__setitem__("cors_origins", ["*"]),
"protected route portal cannot allow wildcard CORS",
)
def test_equivalent_route_paths_are_ambiguous(self):
self.assert_invalid(lambda raw: raw["routes"].append({
"name": "duplicate", "path": "/dist/", "artifact": "portal",
"access": {"mode": "protected", "middleware": "authentik-forwardauth"},
}), "ambiguous")
def test_catch_all_is_required(self):
self.assert_invalid(lambda raw: raw["routes"].__setitem__(0, {
**raw["routes"][0], "path": "/portal"
}), "catch-all")
def test_public_catch_all_is_rejected_when_any_route_is_protected(self):
def mutate(raw):
raw["routes"][0]["access"] = {"mode": "public"}
raw["routes"][1]["access"] = {
"mode": "protected", "middleware": "authentik-forwardauth"
}
raw["artifacts"][1]["cors_origins"] = []
raw["artifacts"][0]["cache"]["rules"][0]["cache_control"] = (
"public, max-age=0, must-revalidate"
)
for rule in raw["artifacts"][1]["cache"]["rules"]:
rule["cache_control"] = "private, no-store"
self.assert_invalid(mutate, "public '/' catch-all")
def test_protected_route_requires_middleware(self):
self.assert_invalid(
lambda raw: raw["routes"][0].__setitem__("access", {"mode": "protected"}),
"middleware is required",
)
def test_bucket_cannot_cross_access_boundary(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0]["publish"].__setitem__("bucket", "baseline-dist"),
"protected and public",
)
def test_publication_credentials_cannot_be_reused(self):
def mutate(raw):
raw["artifacts"][0]["publish"]["credentials"] = copy.deepcopy(
raw["artifacts"][1]["publish"]["credentials"]
)
self.assert_invalid(mutate, "publication credential DIST_S3_ACCESS_KEY is reused")
def test_cache_directive_contradiction_is_rejected(self):
self.assert_invalid(
lambda raw: raw["artifacts"][1]["cache"]["rules"][1].__setitem__(
"cache_control", "public, max-age=31536000, immutable, must-revalidate"
),
"immutable requires",
)
def test_cache_policy_cannot_nest_below_immutable_path(self):
def mutate(raw):
raw["artifacts"][1]["cache"]["rules"].append({
"path": "releases/candidates",
"cache_control": "public, max-age=0, must-revalidate",
})
self.assert_invalid(mutate, "cannot nest another policy under immutable /releases")
def test_public_cache_is_rejected_on_protected_route(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0]["cache"]["rules"][0].__setitem__(
"cache_control", "public, max-age=0, must-revalidate"
),
"protected route portal",
)
def test_legacy_and_split_fields_cannot_mix(self):
self.assert_invalid(lambda raw: raw.__setitem__("type", "static"), "cannot be mixed")
def test_unknown_split_field_is_rejected(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0].__setitem__("storage_bucket", "typo"),
"unknown fields: storage_bucket",
)
def test_backend_authority_and_endpoint_are_derived(self):
for field, value in (
("endpoint", "https://attacker.example"),
("website_authority", "internal-api.default.svc.k8s.sjc001.fritzlab.net"),
):
with self.subTest(field=field):
self.assert_invalid(
lambda raw, field=field, value=value: raw["artifacts"][0]["publish"].__setitem__(field, value),
f"unknown fields: {field}",
)
def test_publication_credentials_use_dedicated_matched_names(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0]["publish"]["credentials"].__setitem__(
"access_key_env", "CI_BOT_TOKEN"
),
"must be a matched <NAME>_S3_ACCESS_KEY",
)
def test_resolved_build_inputs_must_be_pairwise_disjoint(self):
raw = copy.deepcopy(self.raw)
raw["artifacts"][1]["content_dir"] = ""
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "portal" / "build").mkdir(parents=True)
with self.assertRaisesRegex(ConfigError, "build inputs overlap after resolution"):
validate_artifact_inputs(root, cfg)
def test_descendant_symlink_cannot_cross_artifact_boundary(self):
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "dist").mkdir()
(root / "portal" / "build").mkdir(parents=True)
(root / "portal" / "build" / "private.txt").write_text("private")
(root / "dist" / "portal-link").symlink_to(root / "portal" / "build")
with self.assertRaisesRegex(ConfigError, "build input contains symlink"):
validate_artifact_inputs(root, cfg)
def test_artifact_root_symlink_is_rejected_before_resolution(self):
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "dist-real").mkdir()
(root / "dist").symlink_to(root / "dist-real")
(root / "portal" / "build").mkdir(parents=True)
with self.assertRaisesRegex(ConfigError, "content_dir contains symlink component"):
validate_artifact_inputs(root, cfg)
class GenerationTests(unittest.TestCase):
def render(self, raw):
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
tmp = tempfile.TemporaryDirectory()
root = Path(tmp.name)
app_dir = root / "app"
manifests = app_dir / "manifests"
app_dir.mkdir(parents=True)
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg
)
files = {path.relative_to(app_dir).as_posix(): path.read_text()
for path in sorted(app_dir.rglob("*.yaml"))}
return tmp, app_dir, files
def test_split_fixture_generates_per_route_resources_and_one_certificate(self):
tmp, _, files = self.render(fixture("split-site.yaml"))
self.addCleanup(tmp.cleanup)
self.assertEqual({
"app.yaml", "manifests/certificate.yaml", "manifests/ingress-distributions.yaml",
"manifests/ingress-portal.yaml", "manifests/kustomization.yaml",
"manifests/service-distributions.yaml", "manifests/service-portal.yaml",
"site-publish-history.yaml",
}, set(files))
for content in files.values():
self.assertIsNotNone(yaml.safe_load(content))
self.assertIn("path: /dist", files["manifests/ingress-distributions.yaml"])
self.assertNotIn("authentik-forwardauth", files["manifests/ingress-distributions.yaml"])
self.assertIn("authentik-forwardauth@file", files["manifests/ingress-portal.yaml"])
self.assertIn('service.passhostheader: "false"', files["manifests/service-portal.yaml"])
self.assertNotIn("passhostheader", files["manifests/ingress-portal.yaml"])
self.assertIn("baseline-dist.web.sjc001.fritzlab.net", files["manifests/service-distributions.yaml"])
def test_yaml_ambiguous_artifact_name_stays_a_string_annotation(self):
raw = fixture("split-site.yaml")
raw["artifacts"][0]["name"] = "yes"
raw["routes"][0]["artifact"] = "yes"
tmp, _, files = self.render(raw)
self.addCleanup(tmp.cleanup)
ingress = yaml.safe_load(files["manifests/ingress-portal.yaml"])
self.assertEqual(
"yes", ingress["metadata"]["annotations"]["site-publish.fritzlab.net/artifact"],
)
def test_generation_is_deterministic_when_input_lists_are_reversed(self):
raw = fixture("split-site.yaml")
first_tmp, _, first = self.render(raw)
self.addCleanup(first_tmp.cleanup)
raw["artifacts"].reverse()
raw["routes"].reverse()
second_tmp, _, second = self.render(raw)
self.addCleanup(second_tmp.cleanup)
self.assertEqual(first, second)
def test_stale_route_manifests_are_removed(self):
raw = fixture("split-site.yaml")
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp) / "app"
manifests = app_dir / "manifests"
manifests.mkdir(parents=True)
stale = manifests / "ingress-removed.yaml"
stale.write_text("stale\n")
deploy.render_site_manifests("baseline.fritzlab.net", ROOT, app_dir, manifests, cfg)
self.assertFalse(stale.exists())
def test_legacy_names_and_garage_s3_target_are_preserved(self):
tmp, app_dir, files = self.render(fixture("legacy-site.yaml"))
self.addCleanup(tmp.cleanup)
self.assertIn("manifests/service.yaml", files)
self.assertIn("manifests/ingress.yaml", files)
self.assertIn("garage-s3.storage.svc.k8s.sjc001.fritzlab.net", files["manifests/service.yaml"])
self.assertNotIn("passhostheader", files["manifests/ingress.yaml"])
self.assertNotIn("site-publish.fritzlab.net", files["manifests/ingress.yaml"])
self.assertEqual({}, deploy.previous_route_contracts(app_dir))
class BuildTests(unittest.TestCase):
def test_artifacts_build_independently_without_clobbering_siblings(self):
raw = fixture("split-site.yaml")
for artifact in raw["artifacts"]:
artifact["tidy"] = False
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "dist").mkdir()
(root / "dist" / "bundle.js").write_text("bundle")
(root / "portal" / "build").mkdir(parents=True)
(root / "portal" / "build" / "index.html").write_text("portal")
for artifact in cfg["artifacts"]:
build.build_artifact(root, artifact)
self.assertEqual(
"bundle", (root / ".site-publish/distributions/html/bundle.js").read_text()
)
self.assertEqual(
"portal", (root / ".site-publish/portal/html/index.html").read_text()
)
class PublishingTests(unittest.TestCase):
def test_cors_policy_is_reconciled_as_read_only_browser_access(self):
captured = {}
def capture(command, **_kwargs):
config_path = command[command.index("--cors-configuration") + 1]
captured["command"] = command
captured["config"] = json.loads(Path(config_path.removeprefix("file://")).read_text())
with patch.object(deploy, "run", side_effect=capture):
deploy.configure_cors(
"baseline-dist", ["*"], "http://garage-s3.storage.svc:3900", {}
)
self.assertIn("put-bucket-cors", captured["command"])
self.assertEqual(["GET", "HEAD"], captured["config"]["CORSRules"][0]["AllowedMethods"])
self.assertEqual(["*"], captured["config"]["CORSRules"][0]["AllowedOrigins"])
def test_empty_cors_policy_removes_stale_bucket_cors(self):
with patch.object(deploy, "run") as request:
deploy.configure_cors(
"baseline-catalogue", [], "http://garage-s3.storage.svc:3900", {}
)
self.assertIn("delete-bucket-cors", request.call_args.args[0])
def test_cors_reconciliation_restores_prior_policies_on_failure(self):
artifacts = [
{
"bucket": "first", "cors_origins": ["https://new.example"],
"s3_endpoint": "http://garage-s3.storage.svc:3900", "credentials": {},
},
{
"bucket": "second", "cors_origins": [],
"s3_endpoint": "http://garage-s3.storage.svc:3900", "credentials": {},
},
]
prior = [
{"CORSRules": [{"AllowedOrigins": ["https://old.example"]}]},
None,
]
writes = []
def write(bucket, config, *_args):
writes.append((bucket, config))
if bucket == "second" and len(writes) == 2:
raise RuntimeError("write failed")
with patch.object(deploy, "publication_aws_env", return_value={}), patch.object(
deploy, "get_cors_configuration", side_effect=prior,
) as read, patch.object(deploy, "set_cors_configuration", side_effect=write), \
self.assertRaisesRegex(RuntimeError, "write failed"):
deploy.reconcile_artifact_cors(artifacts)
self.assertEqual(2, read.call_count)
self.assertEqual(
[
("first", {"CORSRules": [{
"AllowedOrigins": ["https://new.example"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 3600,
}]}),
("second", None),
("second", None),
("first", prior[0]),
],
writes,
)
def test_apps_clone_never_places_token_in_argv_or_log(self):
calls = []
secret = "clone-secret-must-not-appear"
with patch.dict(os.environ, {"CI_BOT_USER": "ci-bot"}, clear=False), \
patch.object(utils, "run", side_effect=lambda command, **kwargs: calls.append((command, kwargs))), \
patch.object(utils.shutil, "rmtree"), redirect_stdout(io.StringIO()) as output:
utils.clone_apps(secret)
self.assertNotIn(secret, output.getvalue())
for command, kwargs in calls:
self.assertNotIn(secret, " ".join(command))
self.assertNotIn(secret, kwargs.get("display", ""))
def test_askpass_authenticates_clone_and_push_round_trip(self):
token = "round-trip-token"
expected = "Basic " + base64.b64encode(f"ci-bot:{token}".encode()).decode()
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
bare = root / "repo.git"
seed = root / "seed"
checkout = root / "checkout"
subprocess.run(["git", "init", "--bare", "--initial-branch=main", str(bare)], check=True,
stdout=subprocess.DEVNULL)
subprocess.run(["git", "-C", str(bare), "config", "http.receivepack", "true"], check=True)
subprocess.run(["git", "init", "--initial-branch=main", str(seed)], check=True,
stdout=subprocess.DEVNULL)
subprocess.run(["git", "-C", str(seed), "config", "user.name", "Test"], check=True)
subprocess.run(["git", "-C", str(seed), "config", "user.email", "test@example.invalid"], check=True)
(seed / "README.md").write_text("seed\n")
subprocess.run(["git", "-C", str(seed), "add", "README.md"], check=True)
subprocess.run(["git", "-C", str(seed), "commit", "-m", "seed"], check=True,
stdout=subprocess.DEVNULL)
subprocess.run(["git", "-C", str(seed), "push", str(bare), "main"], check=True,
stdout=subprocess.DEVNULL)
handler = type("GitHandler", (AuthenticatedGitHandler,), {
"project_root": root, "expected_authorization": expected,
})
server = IPv6GitHTTPServer(("::1", 0), handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
url = f"http://ci-bot@[::1]:{server.server_port}/repo.git"
with patch.dict(os.environ, {"NO_PROXY": "::1,[::1]", "no_proxy": "::1,[::1]"}, clear=False):
subprocess.run(
["git", "clone", url, str(checkout)], env=utils.git_auth_env(token),
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)
subprocess.run(["git", "-C", str(checkout), "config", "user.name", "Test"], check=True)
subprocess.run(["git", "-C", str(checkout), "config", "user.email", "test@example.invalid"],
check=True)
(checkout / "roundtrip.txt").write_text("authenticated\n")
utils.commit_and_push(checkout, "authenticated round trip", token)
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
result = subprocess.run(
["git", "--git-dir", str(bare), "show", "main:roundtrip.txt"],
check=True, text=True, capture_output=True,
)
self.assertEqual("authenticated\n", result.stdout)
def test_cache_headers_credentials_and_route_prefix_are_separate(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
html = root / artifact["build_dir"]
(html / "releases").mkdir(parents=True)
(html / "channels").mkdir()
(html / "releases" / "1.0.js").write_text("release")
(html / "channels" / "stable.json").write_text("channel")
commands, events = [], []
def capture(command, **kwargs):
commands.append((command, kwargs["env"]))
events.append("mutable")
def publish_immutable(*_args):
events.append("immutable")
secret = "secret-must-not-appear"
output = io.StringIO()
with patch.dict(os.environ, {
"DIST_S3_ACCESS_KEY": "dist-key", "DIST_S3_SECRET_KEY": secret
}, clear=False), patch.object(deploy, "run", side_effect=capture), \
patch.object(deploy, "publish_immutable_rule", side_effect=publish_immutable) as immutable_publish, \
redirect_stdout(output):
deploy.publish_route_immutables(artifact, route, root)
deploy.s3_sync(artifact, route, root)
self.assertTrue(all(secret not in " ".join(command) for command, _ in commands))
self.assertEqual("immutable", events[0])
self.assertNotIn(secret, output.getvalue())
self.assertTrue(all(call_env["AWS_ACCESS_KEY_ID"] == "dist-key" for _, call_env in commands))
self.assertTrue(all("DIST_S3_SECRET_KEY" not in call_env for _, call_env in commands))
rendered = [" ".join(command) for command, _ in commands]
self.assertIn("s3://baseline-dist/dist/", rendered[0])
self.assertIn("releases/*", rendered[0])
self.assertNotIn("--delete", rendered[0])
self.assertIn("--delete", rendered[1])
self.assertTrue(all("s3://baseline-dist/dist/" in command for command in rendered[1:]))
self.assertTrue(any("channels/" in command and
"public, max-age=0, must-revalidate" in command
for command in rendered))
immutable_publish.assert_called_once()
self.assertEqual(
"public, max-age=31536000, immutable",
immutable_publish.call_args.args[2]["cache_control"],
)
def test_root_move_preserves_only_actual_retired_immutable_prefix(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
route = {**next(item for item in cfg["routes"] if item["artifact"] == "distributions"),
"path": "/"}
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
html = root / artifact["build_dir"]
(html / "releases").mkdir(parents=True)
(html / "channels").mkdir()
(html / "docs" / "releases").mkdir(parents=True)
(html / "channels" / "stable.json").write_text("channel")
(html / "docs" / "releases" / "index.html").write_text("mutable")
commands = []
with patch.dict(os.environ, {
"DIST_S3_ACCESS_KEY": "dist-key", "DIST_S3_SECRET_KEY": "dist-secret"
}, clear=False), patch.object(
deploy, "run", side_effect=lambda command, **_: commands.append(command)
):
deploy.s3_sync(artifact, route, root, previous_contract={
"path": "/foo", "access": "public", "artifact": "distributions",
"immutable_paths": ["foo/releases"],
})
rendered = [" ".join(command) for command in commands]
self.assertTrue(all("foo/releases/*" in command for command in rendered[:2]))
self.assertTrue(all("*/releases/*" not in command for command in rendered))
def test_protected_bucket_cannot_become_public_across_deployments(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
previous = {
"baseline-dist": {
"path": "/dist", "access": "protected", "artifact": "old-name",
"immutable_paths": ["dist/releases"],
}
}
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(cfg, previous)
renamed = copy.deepcopy(cfg)
artifact = next(item for item in renamed["artifacts"] if item["name"] == "distributions")
artifact["name"] = "downloads"
route = next(item for item in renamed["routes"] if item["artifact"] == "distributions")
route["artifact"] = "downloads"
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(renamed, previous)
previous = {"retired-protected-bucket": previous["baseline-dist"]}
deploy.validate_route_migrations(cfg, previous)
def test_protected_split_bucket_cannot_become_legacy_public(self):
cfg = normalize_site_config(fixture("legacy-site.yaml"), "baseline.fritzlab.net")
previous = {
"baseline.fritzlab.net": {
"path": "/portal", "access": "protected", "artifact": "portal",
"immutable_paths": [],
}
}
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(cfg, previous)
def test_removed_immutable_rule_preserves_prior_keys(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
with tempfile.TemporaryDirectory() as tmp:
html = Path(tmp)
filters = deploy.retired_immutable_filters(artifact, route, html, {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
})
self.assertEqual(["--exclude", "releases/*"], filters)
(html / "releases").mkdir()
(html / "releases" / "replacement.js").write_text("mutable")
with self.assertRaisesRegex(RuntimeError, "collides with retired immutable"):
deploy.retired_immutable_filters(artifact, route, html, {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
})
def test_removed_immutable_rule_remains_in_next_manifest_history(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
previous = {
"baseline-dist": {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
}
}
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp) / "app"
manifests = app_dir / "manifests"
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, previous,
)
first = deploy.previous_route_contracts(app_dir)
self.assertEqual(["dist/releases"], first["baseline-dist"]["immutable_paths"])
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, first,
)
second = deploy.previous_route_contracts(app_dir)
self.assertEqual(first, second)
def test_protected_split_bucket_survives_absence_and_blocks_legacy(self):
raw = fixture("split-site.yaml")
raw["artifacts"] = [
item for item in raw["artifacts"] if item["name"] == "distributions"
]
raw["routes"] = [
item for item in raw["routes"] if item["artifact"] == "distributions"
]
raw["routes"][0]["path"] = "/"
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
previous = {
"baseline.fritzlab.net": {
"path": "/", "access": "protected", "artifact": "portal",
"immutable_paths": [],
}
}
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp) / "app"
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, app_dir / "manifests", cfg, previous,
)
retained = deploy.previous_route_contracts(app_dir)
self.assertEqual(
previous["baseline.fritzlab.net"], retained["baseline.fritzlab.net"],
)
legacy = normalize_site_config(
fixture("legacy-site.yaml"), "baseline.fritzlab.net",
)
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(legacy, retained)
def test_malformed_persistent_history_fails_closed(self):
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp)
(app_dir / "site-publish-history.yaml").write_text(
"version: 1\nbuckets:\n bucket:\n path: /\n"
" access: public\n artifact: site\n"
" immutable_paths: [../releases]\n"
)
with self.assertRaisesRegex(RuntimeError, "invalid site-publish route history"):
deploy.previous_route_contracts(app_dir)
def test_retired_immutable_history_survives_a_route_move(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
route["path"] = "/"
previous = {
"baseline-dist": {
"path": "/foo", "access": "public", "artifact": "distributions",
"immutable_paths": ["foo/releases"],
}
}
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
app_dir = root / "app"
manifests = app_dir / "manifests"
app_dir.mkdir()
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, previous,
)
following = deploy.previous_route_contracts(app_dir)
self.assertEqual(["foo/releases"], following["baseline-dist"]["immutable_paths"])
html = root / "html"
html.mkdir()
first_filters = deploy.retired_immutable_filters(
artifact, route, html, previous["baseline-dist"],
)
following_filters = deploy.retired_immutable_filters(
artifact, route, html, following["baseline-dist"],
)
self.assertEqual(["--exclude", "foo/releases/*"], first_filters)
self.assertEqual(first_filters, following_filters)
def test_later_route_immutable_failure_stops_all_mutable_publication(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
with patch.object(deploy, "validate_publication_environment"), \
patch.object(deploy, "validate_artifact_output"), \
patch.object(deploy, "clone_apps", return_value=root / "apps"), patch.object(
deploy, "publish_route_immutables",
side_effect=[None, RuntimeError("immutable failed")],
) as immutable_publish, patch.object(
deploy, "reconcile_artifact_cors"
) as cors_reconcile, patch.object(deploy, "s3_sync") as mutable_sync, \
self.assertRaisesRegex(
RuntimeError, "immutable failed"
):
deploy.deploy_static("baseline", root, root, "token", cfg)
self.assertEqual(2, immutable_publish.call_count)
cors_reconcile.assert_not_called()
mutable_sync.assert_not_called()
def test_all_cors_policies_complete_before_mutable_publication(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
events = []
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
apps = root / "apps"
apps.mkdir()
with patch.object(deploy, "validate_publication_environment"), patch.object(
deploy, "validate_artifact_output"
), patch.object(
deploy, "publish_route_immutables", side_effect=lambda *_args: events.append("immutable")
), patch.object(
deploy, "reconcile_artifact_cors", side_effect=lambda *_args: events.append("cors")
), patch.object(
deploy, "s3_sync", side_effect=lambda *_args: events.append("mutable")
), patch.object(deploy, "clone_apps", return_value=apps), patch.object(
deploy, "render_site_manifests"
), patch.object(deploy, "commit_and_push"):
deploy.deploy_static("baseline", root, root, "token", cfg)
self.assertEqual(
["immutable", "immutable", "cors", "mutable", "mutable"], events
)
def test_absent_artifact_is_detected_before_publish(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp, redirect_stderr(io.StringIO()), \
self.assertRaises(SystemExit):
deploy.validate_artifact_output(Path(tmp), cfg["artifacts"][0])
def test_absent_cache_prefix_is_detected(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
with tempfile.TemporaryDirectory() as tmp:
html = Path(tmp) / artifact["build_dir"]
html.mkdir(parents=True)
(html / "index.html").write_text("content")
with redirect_stderr(io.StringIO()), self.assertRaises(SystemExit):
deploy.validate_artifact_output(Path(tmp), artifact)
class ImmutablePublicationTests(unittest.TestCase):
CACHE = "public, max-age=31536000, immutable"
def result(self, returncode, stdout="", stderr=""):
return subprocess.CompletedProcess([], returncode, stdout, stderr)
def key_and_digests(self, source):
content_type = "text/javascript"
content_digest, publication_digest = deploy._immutable_digests(
source, self.CACHE, content_type,
)
return f"dist/releases/release-{publication_digest}.js", content_digest, publication_digest
def test_new_key_uses_content_address_and_digest_metadata(self):
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "release.js"
source.write_text("fixed release")
key, content_digest, publication_digest = self.key_and_digests(source)
calls = []
def capture(args, _env):
calls.append(args)
return self.result(1, stderr="404 Not Found") if len(calls) == 1 else self.result(0, "{}")
with patch.object(deploy, "_aws_capture", side_effect=capture):
created = deploy.publish_immutable_file(
"http://garage-s3.storage.svc:3900", "dist-bucket",
key, source, self.CACHE, {},
)
self.assertTrue(created)
put = calls[1]
self.assertEqual("put-object", put[4])
self.assertNotIn("--if-none-match", put)
self.assertEqual(
f"sha256={content_digest},publication-sha256={publication_digest}",
put[put.index("--metadata") + 1],
)
def test_identical_retry_converges_without_put(self):
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "release.js"
source.write_text("fixed release")
key, content_digest, publication_digest = self.key_and_digests(source)
head = json.dumps({
"Metadata": {"sha256": content_digest, "publication-sha256": publication_digest},
"CacheControl": self.CACHE,
"ContentType": "text/javascript",
})
with patch.object(deploy, "_aws_capture", return_value=self.result(0, head)) as request:
created = deploy.publish_immutable_file(
"http://garage-s3.storage.svc:3900", "dist-bucket",
key, source, self.CACHE, {},
)
self.assertFalse(created)
self.assertEqual(1, request.call_count)
def test_changed_immutable_key_is_refused(self):
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "release.js"
source.write_text("changed release")
key, _, _ = self.key_and_digests(source)
head = json.dumps({"Metadata": {"sha256": "different"}, "CacheControl": self.CACHE})
with patch.object(deploy, "_aws_capture", return_value=self.result(0, head)), \
self.assertRaisesRegex(RuntimeError, "immutable object differs"):
deploy.publish_immutable_file(
"http://garage-s3.storage.svc:3900", "dist-bucket",
key, source, self.CACHE, {},
)
def test_immutable_key_without_publication_digest_is_refused_before_s3(self):
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "release.js"
source.write_text("fixed release")
with patch.object(deploy, "_aws_capture") as request, \
self.assertRaisesRegex(RuntimeError, "must contain its one publication SHA-256"):
deploy.publish_immutable_file(
"http://garage-s3.storage.svc:3900", "dist-bucket",
"dist/releases/release.js", source, self.CACHE, {},
)
request.assert_not_called()
if __name__ == "__main__":
unittest.main()