Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8ec44fd4aa | ||
|
|
85a0b41380 | ||
|
|
19fb4e43ab | ||
|
|
7b824a61ca | ||
|
|
fb2e440bbb | ||
|
|
5261b9b99f | ||
|
|
5c2630b972 |
@@ -0,0 +1,14 @@
|
|||||||
|
name: Test
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
jobs:
|
||||||
|
contract:
|
||||||
|
runs-on: fritzlab
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Install test dependencies
|
||||||
|
run: python3 -m pip install --quiet --break-system-packages jinja2 pyyaml
|
||||||
|
- name: Run contract tests
|
||||||
|
run: python3 -m unittest discover -s tests -v
|
||||||
@@ -1,9 +1,11 @@
|
|||||||
# action/site-publish
|
# action/site-publish
|
||||||
|
|
||||||
Composite Gitea Action that publishes a **static-content** website to the
|
Composite Gitea Action that publishes one or more **static-content** artifacts
|
||||||
fritzlab k8s cluster. Supports `static`, `hugo`, and `mkdocs`. Content goes
|
to one hostname. Each split surface owns its build input, Garage bucket,
|
||||||
to a Garage S3 bucket; Traefik fronts the bucket via an `ExternalName`
|
publication credential environment variables, cache rules, Service, Ingress,
|
||||||
Service with cert-manager TLS.
|
route prefix, and access middleware. The hostname shares one Certificate.
|
||||||
|
`static`, `hugo`, and `mkdocs` builds are supported; a prebuilt Docusaurus
|
||||||
|
output is a `static` artifact.
|
||||||
|
|
||||||
> **Containerized web apps (Dockerfile-based) are NOT handled here.** Use the
|
> **Containerized web apps (Dockerfile-based) are NOT handled here.** Use the
|
||||||
> standard image-producer chain instead:
|
> standard image-producer chain instead:
|
||||||
@@ -16,9 +18,10 @@ Service with cert-manager TLS.
|
|||||||
> for the canonical example. site-publish errors out explicitly if
|
> for the canonical example. site-publish errors out explicitly if
|
||||||
> `site.yaml` has `type: docker`.
|
> `site.yaml` has `type: docker`.
|
||||||
|
|
||||||
## Convention
|
## Single-surface compatibility
|
||||||
|
|
||||||
Bucket name = repo name = canonical domain. Sibling hostnames (e.g. `www.`,
|
Existing `site.yaml` files remain the `single-surface-v1` compatibility
|
||||||
|
contract. Bucket name = repo name = canonical domain. Sibling hostnames (e.g. `www.`,
|
||||||
`ipv6.`) are declared as `aliases:` in `site.yaml` — the action registers each
|
`ipv6.`) are declared as `aliases:` in `site.yaml` — the action registers each
|
||||||
as a Garage `globalAlias` on the bucket and adds it to the Ingress + Certificate
|
as a Garage `globalAlias` on the bucket and adds it to the Ingress + Certificate
|
||||||
on every deploy. Manual edits to manifests in the apps repo are clobbered;
|
on every deploy. Manual edits to manifests in the apps repo are clobbered;
|
||||||
@@ -32,6 +35,133 @@ Scaffold a new site (handles repo creation + Garage bucket):
|
|||||||
./new-site.sh --name my-site.vino.network --domain my-site.vino.network --type static
|
./new-site.sh --name my-site.vino.network --domain my-site.vino.network --type static
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The compatibility path still writes `build/html`, uploads with
|
||||||
|
`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, and renders `service.yaml` plus
|
||||||
|
`ingress.yaml`. Its only behavior change is the required root-cause repair:
|
||||||
|
the website Service now targets the data-only `garage-s3` Service.
|
||||||
|
|
||||||
|
## Split-surface contract
|
||||||
|
|
||||||
|
Use `artifacts` and `routes` together. This example expresses an authenticated
|
||||||
|
prebuilt portal at `/` and public bundles at `/dist`; it is illustrative and
|
||||||
|
the schema has no Baseline-specific field.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
domain: baseline.fritzlab.net
|
||||||
|
artifacts:
|
||||||
|
- name: distributions
|
||||||
|
type: static
|
||||||
|
content_dir: dist
|
||||||
|
publish:
|
||||||
|
bucket: baseline-dist
|
||||||
|
credentials:
|
||||||
|
access_key_env: DIST_S3_ACCESS_KEY
|
||||||
|
secret_key_env: DIST_S3_SECRET_KEY
|
||||||
|
cache:
|
||||||
|
rules:
|
||||||
|
- path: /
|
||||||
|
cache_control: public, max-age=0, must-revalidate
|
||||||
|
- path: releases
|
||||||
|
cache_control: public, max-age=31536000, immutable
|
||||||
|
- path: channels
|
||||||
|
cache_control: public, max-age=0, must-revalidate
|
||||||
|
- name: portal
|
||||||
|
type: static
|
||||||
|
content_dir: portal/build
|
||||||
|
publish:
|
||||||
|
bucket: baseline-portal
|
||||||
|
credentials:
|
||||||
|
access_key_env: PORTAL_S3_ACCESS_KEY
|
||||||
|
secret_key_env: PORTAL_S3_SECRET_KEY
|
||||||
|
cache:
|
||||||
|
rules:
|
||||||
|
- path: /
|
||||||
|
cache_control: private, no-store
|
||||||
|
routes:
|
||||||
|
- name: distributions
|
||||||
|
path: /dist
|
||||||
|
artifact: distributions
|
||||||
|
access:
|
||||||
|
mode: public
|
||||||
|
- name: portal
|
||||||
|
path: /
|
||||||
|
artifact: portal
|
||||||
|
access:
|
||||||
|
mode: protected
|
||||||
|
middleware: authentik-forwardauth
|
||||||
|
```
|
||||||
|
|
||||||
|
The caller supplies each declared credential name as an environment variable
|
||||||
|
on the action step. Names must be matched `<NAME>_S3_ACCESS_KEY` and
|
||||||
|
`<NAME>_S3_SECRET_KEY` pairs; arbitrary environment variables cannot become
|
||||||
|
publication credentials. Values pass to `aws` only through its environment and
|
||||||
|
never appear in a logged command or process argument.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- uses: https://code.fritzlab.net/action/site-publish@v1
|
||||||
|
with:
|
||||||
|
token: ${{ secrets.CI_BOT_TOKEN }}
|
||||||
|
env:
|
||||||
|
DIST_S3_ACCESS_KEY: ${{ secrets.DIST_S3_ACCESS_KEY }}
|
||||||
|
DIST_S3_SECRET_KEY: ${{ secrets.DIST_S3_SECRET_KEY }}
|
||||||
|
PORTAL_S3_ACCESS_KEY: ${{ secrets.PORTAL_S3_ACCESS_KEY }}
|
||||||
|
PORTAL_S3_SECRET_KEY: ${{ secrets.PORTAL_S3_SECRET_KEY }}
|
||||||
|
```
|
||||||
|
|
||||||
|
Routes are normalized and rendered longest-prefix first. Split mode requires
|
||||||
|
one `/` catch-all so unmatched paths have an explicit access policy. If any
|
||||||
|
route is protected, that catch-all must also be protected. Every artifact must
|
||||||
|
belong to exactly one route and bucket; protected and public routes cannot
|
||||||
|
reuse a bucket. A protected route requires an existing file-provider access
|
||||||
|
middleware. Public routes cannot declare one.
|
||||||
|
|
||||||
|
Every cache policy requires a `/` default. More-specific cache paths override
|
||||||
|
it, are reapplied in deterministic prefix order, and must exist in the built
|
||||||
|
artifact. Contradictory directives (`public` plus `private`, `immutable` plus
|
||||||
|
revalidation, or `no-store` plus a positive max-age) are rejected. Protected
|
||||||
|
artifacts require `private` or `no-store` and cannot emit `public`.
|
||||||
|
Metadata restamping transfers each artifact once even on a no-op publication;
|
||||||
|
that is the cost of making policy changes effective on unchanged Garage objects.
|
||||||
|
An immutable cache path is excluded from sync and deletion. Every object key in
|
||||||
|
that path must contain exactly one full publication SHA-256, calculated over its
|
||||||
|
cache policy, content type, and bytes. That content address makes concurrent
|
||||||
|
writes identical even though Garage v2.2.0 has no conditional destination
|
||||||
|
write. An identical retry converges; a changed object, missing digest metadata,
|
||||||
|
wrong address, or nested policy under that immutable prefix fails publication.
|
||||||
|
Every immutable target across every artifact is validated and published before
|
||||||
|
any route's mutable objects change.
|
||||||
|
Mutable default and override partitions receive their final cache policy before
|
||||||
|
the matching prefix-scoped stale deletion, so publication never exposes a
|
||||||
|
provisional cache policy or a pointer to a missing immutable target.
|
||||||
|
An append-only `site-publish-history.json` beside each generated site retains every bucket's access
|
||||||
|
class and absolute immutable prefixes, including removed routes and rules. When a route move places
|
||||||
|
a retired prefix inside the new sync scope, that subtree is excluded; a current-file collision
|
||||||
|
fails publication. Protected access remains sticky across artifact renames and legacy mode, so
|
||||||
|
publishing the same artifact publicly requires a new bucket. Decommissioning removes the live
|
||||||
|
application and manifests while retaining this history because its Garage bucket is not purged.
|
||||||
|
|
||||||
|
Artifact input directories must be pairwise disjoint after filesystem
|
||||||
|
resolution. Publication stops before build or upload if one contains another or
|
||||||
|
escapes the repository. Symlinked roots, components, and descendants are also rejected, preventing
|
||||||
|
protected input from entering a public artifact through dereference. Split
|
||||||
|
storage endpoints are pinned to Garage, and each website
|
||||||
|
authority is derived from its bucket; a site cannot expose an arbitrary backend.
|
||||||
|
|
||||||
|
Each split route gets a bucket-specific `<bucket>.web.sjc001.fritzlab.net`
|
||||||
|
ExternalName Service annotated to disable pass-host-header and a separate Ingress. Route
|
||||||
|
Ingresses share the hostname's certificate Secret. The access middleware and
|
||||||
|
Garage bucket/key must already exist; the publisher doesn't create identity
|
||||||
|
providers or credentials.
|
||||||
|
|
||||||
|
### Migrating a site
|
||||||
|
|
||||||
|
Leave an existing single-surface file unchanged until a real second surface
|
||||||
|
exists. Then build every artifact before this action, move the old fields into
|
||||||
|
an artifact, declare a route for every artifact, give each bucket a separately
|
||||||
|
scoped key, and set the route access/cache contract. Run the repository tests
|
||||||
|
and inspect generated Apps changes. Removing a route removes its generated
|
||||||
|
Service and Ingress on the next render; bucket deletion remains manual.
|
||||||
|
|
||||||
Or do it manually. `site.yaml`:
|
Or do it manually. `site.yaml`:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
@@ -75,11 +205,11 @@ jobs:
|
|||||||
garage-admin-token: ${{ secrets.GARAGE_ADMIN_TOKEN }}
|
garage-admin-token: ${{ secrets.GARAGE_ADMIN_TOKEN }}
|
||||||
```
|
```
|
||||||
|
|
||||||
DNS: subdomains of `vino.network` are covered by the wildcard CNAME to
|
DNS: subdomains of `vino.network` are covered by the wildcard CNAME to the
|
||||||
`traefik.edge.svc…`. For other zones, add an explicit CNAME:
|
public gateway. For other zones, add an explicit CNAME:
|
||||||
|
|
||||||
```
|
```
|
||||||
my-site.fritzlab.net 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net.
|
my-site.fritzlab.net 300 IN CNAME gateway.sjc001.fritzlab.net.
|
||||||
```
|
```
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
@@ -87,10 +217,10 @@ my-site.fritzlab.net 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net.
|
|||||||
| Input | Required | Default | Description |
|
| Input | Required | Default | Description |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `token` | yes | | Gitea token for apps repo push |
|
| `token` | yes | | Gitea token for apps repo push |
|
||||||
| `s3-access-key` | yes | | Garage `ci-deploy-key` access key id |
|
| `s3-access-key` | legacy only | | Garage access key id for single-surface sites |
|
||||||
| `s3-secret-key` | yes | | Garage `ci-deploy-key` secret key |
|
| `s3-secret-key` | legacy only | | Garage secret key for single-surface sites |
|
||||||
| `s3-endpoint` | no | `http://garage.storage.svc:3900` | Garage S3 endpoint |
|
| `s3-endpoint` | no | `http://garage-s3.storage.svc:3900` | Legacy Garage S3 endpoint |
|
||||||
| `garage-admin-token` | only if site has `aliases` | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
|
| `garage-admin-token` | legacy aliases only | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
|
||||||
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
|
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
|
||||||
| `username` | no | `ci-bot` | Gitea username |
|
| `username` | no | `ci-bot` | Gitea username |
|
||||||
|
|
||||||
@@ -107,12 +237,11 @@ Org secrets in `websites`: `CI_BOT_TOKEN`, `GARAGE_S3_ACCESS_KEY`,
|
|||||||
```
|
```
|
||||||
push to websites/<repo>
|
push to websites/<repo>
|
||||||
→ CI runs site-publish action
|
→ CI runs site-publish action
|
||||||
→ reads site.yaml, builds content (static copy / hugo / mkdocs), runs tidy
|
→ reads and validates all of site.yaml before publication
|
||||||
→ aws s3 sync → Garage bucket named after the repo
|
→ independently builds each static / Hugo / MkDocs artifact
|
||||||
→ admin API: ensures every alias from site.yaml is a globalAlias on the bucket
|
→ syncs each artifact to its route-owned Garage bucket and prefix
|
||||||
→ renders manifests in fritzlab/apps from templates: ExternalName Service →
|
→ reapplies the artifact's default and longest-prefix cache headers
|
||||||
garage.storage.svc, Traefik Ingress (canonical + aliases), cert-manager
|
→ renders one Service + Ingress per route and one shared Certificate
|
||||||
Certificate (canonical + aliases as SANs), kustomization
|
|
||||||
→ commits + pushes apps repo only if diff is non-empty
|
→ commits + pushes apps repo only if diff is non-empty
|
||||||
→ ArgoCD syncs → site live with TLS
|
→ ArgoCD syncs → site live with TLS
|
||||||
```
|
```
|
||||||
@@ -120,9 +249,10 @@ push to websites/<repo>
|
|||||||
The Ingress + Certificate are re-rendered on every deploy from `site.yaml`.
|
The Ingress + Certificate are re-rendered on every deploy from `site.yaml`.
|
||||||
There is no "first-deploy vs. update" branching — every deploy is idempotent.
|
There is no "first-deploy vs. update" branching — every deploy is idempotent.
|
||||||
|
|
||||||
No nginx pods, no per-site Docker images. Garage matches `Host:` header to
|
No nginx pods, no per-site Docker images. Compatibility sites pass the public
|
||||||
bucket name (or any of its globalAliases), so every site shares a single
|
host to the shared data-only Garage website Service. Split routes disable host
|
||||||
ExternalName target.
|
passing on their Service so Garage receives that artifact's bucket-specific
|
||||||
|
website authority.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
|
|
||||||
|
|||||||
+6
-6
@@ -1,15 +1,15 @@
|
|||||||
name: Publish Site
|
name: Publish Site
|
||||||
description: Build and deploy a static-content site (static, hugo, mkdocs) to Garage S3 with Traefik + cert-manager. Containerized apps should use action/image-build + action/image-push + action/image-deploy.
|
description: Build and deploy one or more routed static-content artifacts to Garage S3 with Traefik and cert-manager.
|
||||||
inputs:
|
inputs:
|
||||||
token:
|
token:
|
||||||
description: Gitea token (ci-bot) for apps repo push and API operations
|
description: Gitea token (ci-bot) for apps repo push and API operations
|
||||||
required: true
|
required: true
|
||||||
s3-access-key:
|
s3-access-key:
|
||||||
description: Garage ci-deploy-key access key id
|
description: Garage access key id (required by the legacy single-surface contract)
|
||||||
required: true
|
required: false
|
||||||
s3-secret-key:
|
s3-secret-key:
|
||||||
description: Garage ci-deploy-key secret access key
|
description: Garage secret access key (required by the legacy single-surface contract)
|
||||||
required: true
|
required: false
|
||||||
s3-endpoint:
|
s3-endpoint:
|
||||||
# Targets garage-s3 (data-only Service) so requests do not round-robin onto
|
# Targets garage-s3 (data-only Service) so requests do not round-robin onto
|
||||||
# the gateway pod, whose emptyDir-backed metadata view intermittently
|
# the gateway pod, whose emptyDir-backed metadata view intermittently
|
||||||
@@ -18,7 +18,7 @@ inputs:
|
|||||||
required: false
|
required: false
|
||||||
default: http://garage-s3.storage.svc:3900
|
default: http://garage-s3.storage.svc:3900
|
||||||
garage-admin-token:
|
garage-admin-token:
|
||||||
description: Garage admin API token (required only when site.yaml has aliases — used to reconcile bucket globalAliases)
|
description: Garage admin API token (required only for legacy aliases — used to reconcile bucket globalAliases)
|
||||||
required: false
|
required: false
|
||||||
garage-admin-endpoint:
|
garage-admin-endpoint:
|
||||||
description: Garage admin API endpoint URL
|
description: Garage admin API endpoint URL
|
||||||
|
|||||||
+2
-2
@@ -165,6 +165,6 @@ echo
|
|||||||
echo "Site created: ${ORG}/${NAME}"
|
echo "Site created: ${ORG}/${NAME}"
|
||||||
echo "First build will trigger on push."
|
echo "First build will trigger on push."
|
||||||
echo
|
echo
|
||||||
echo "DNS: ${DOMAIN} is covered by the *.vino.network wildcard (→ traefik.edge)."
|
echo "DNS: ${DOMAIN} is covered by the *.vino.network wildcard (→ public gateway)."
|
||||||
echo "For a domain outside vino.network, add an explicit CNAME:"
|
echo "For a domain outside vino.network, add an explicit CNAME:"
|
||||||
echo " ${DOMAIN} 300 IN CNAME traefik.edge.svc.k8s.sjc001.fritzlab.net."
|
echo " ${DOMAIN} 300 IN CNAME gateway.sjc001.fritzlab.net."
|
||||||
|
|||||||
+26
-21
@@ -1,25 +1,25 @@
|
|||||||
"""Build phase — content prep for static-content sites."""
|
"""Build each declared static-content artifact independently."""
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from utils import EXCLUDE_FILES, env, parse_site_yaml, run
|
from utils import EXCLUDE_FILES, env, parse_site_yaml, run, validate_artifact_inputs
|
||||||
|
|
||||||
|
|
||||||
def build_static(site_dir, cfg):
|
def build_artifact(site_dir, artifact):
|
||||||
build_dir = site_dir / "build"
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
html_dir = build_dir / "html"
|
if html_dir.parent.exists():
|
||||||
|
shutil.rmtree(html_dir.parent)
|
||||||
|
|
||||||
if build_dir.exists():
|
content_dir = artifact["content_dir"]
|
||||||
shutil.rmtree(build_dir)
|
|
||||||
|
|
||||||
content_dir = cfg["content_dir"]
|
|
||||||
src = site_dir / content_dir if content_dir else site_dir
|
src = site_dir / content_dir if content_dir else site_dir
|
||||||
|
if not src.exists():
|
||||||
|
raise FileNotFoundError(f"artifact {artifact['name']} content_dir not found: {src}")
|
||||||
|
|
||||||
if cfg["type"] == "static":
|
if artifact["type"] == "static":
|
||||||
print(f"Copying static content from {src}")
|
print(f"Copying artifact {artifact['name']} from {src}")
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
tmp_path = Path(tmp) / "html"
|
tmp_path = Path(tmp) / "html"
|
||||||
shutil.copytree(src, tmp_path, dirs_exist_ok=True)
|
shutil.copytree(src, tmp_path, dirs_exist_ok=True)
|
||||||
@@ -29,18 +29,18 @@ def build_static(site_dir, cfg):
|
|||||||
shutil.rmtree(p)
|
shutil.rmtree(p)
|
||||||
elif p.exists():
|
elif p.exists():
|
||||||
p.unlink()
|
p.unlink()
|
||||||
build_dir.mkdir(parents=True)
|
html_dir.parent.mkdir(parents=True)
|
||||||
shutil.move(str(tmp_path), str(html_dir))
|
shutil.move(str(tmp_path), str(html_dir))
|
||||||
|
|
||||||
elif cfg["type"] == "hugo":
|
elif artifact["type"] == "hugo":
|
||||||
print(f"Building Hugo site from {src}")
|
print(f"Building Hugo artifact {artifact['name']} from {src}")
|
||||||
run(f"hugo --source {src} --destination {html_dir}")
|
run(["hugo", "--source", str(src), "--destination", str(html_dir)])
|
||||||
|
|
||||||
elif cfg["type"] == "mkdocs":
|
elif artifact["type"] == "mkdocs":
|
||||||
print(f"Building MkDocs site from {src}")
|
print(f"Building MkDocs artifact {artifact['name']} from {src}")
|
||||||
run(f"cd {src} && mkdocs build -d {html_dir}")
|
run(["mkdocs", "build", "-d", str(html_dir)], cwd=src)
|
||||||
|
|
||||||
if cfg.get("tidy", True):
|
if artifact["tidy"]:
|
||||||
print("Running tidy on HTML files...")
|
print("Running tidy on HTML files...")
|
||||||
for html_file in html_dir.rglob("*.html"):
|
for html_file in html_dir.rglob("*.html"):
|
||||||
subprocess.run(
|
subprocess.run(
|
||||||
@@ -51,7 +51,9 @@ def build_static(site_dir, cfg):
|
|||||||
check=False,
|
check=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
print(f"Build complete — content at {html_dir}")
|
if not any(path.is_file() for path in html_dir.rglob("*")):
|
||||||
|
raise FileNotFoundError(f"artifact {artifact['name']} produced no files in {html_dir}")
|
||||||
|
print(f"Artifact {artifact['name']} complete — content at {html_dir}")
|
||||||
|
|
||||||
|
|
||||||
def cmd_build():
|
def cmd_build():
|
||||||
@@ -62,4 +64,7 @@ def cmd_build():
|
|||||||
print("Site disabled — skipping build")
|
print("Site disabled — skipping build")
|
||||||
return
|
return
|
||||||
|
|
||||||
build_static(site_dir, cfg)
|
validate_artifact_inputs(site_dir, cfg)
|
||||||
|
|
||||||
|
for artifact in cfg["artifacts"]:
|
||||||
|
build_artifact(site_dir, artifact)
|
||||||
|
|||||||
+373
-56
@@ -1,17 +1,18 @@
|
|||||||
"""Deploy phase — S3 sync, manifest rendering, alias reconcile."""
|
"""Deploy phase — S3 sync, manifest rendering, alias reconcile."""
|
||||||
|
|
||||||
|
import fnmatch
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
import shlex
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import tempfile
|
import subprocess
|
||||||
from pathlib import Path
|
from pathlib import Path, PurePosixPath
|
||||||
from urllib.error import HTTPError, URLError
|
from urllib.error import HTTPError, URLError
|
||||||
from urllib.request import Request, urlopen
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
from utils import (
|
from utils import (
|
||||||
DEFAULT_S3_ENDPOINT,
|
|
||||||
GITEA_HOST,
|
|
||||||
NAMESPACE,
|
NAMESPACE,
|
||||||
clone_apps,
|
clone_apps,
|
||||||
commit_and_push,
|
commit_and_push,
|
||||||
@@ -21,51 +22,246 @@ from utils import (
|
|||||||
parse_site_yaml,
|
parse_site_yaml,
|
||||||
render_templates,
|
render_templates,
|
||||||
run,
|
run,
|
||||||
|
validate_artifact_inputs,
|
||||||
)
|
)
|
||||||
|
|
||||||
GARAGE_ADMIN_ENDPOINT = os.environ.get(
|
GARAGE_ADMIN_ENDPOINT = os.environ.get(
|
||||||
"GARAGE_ADMIN_ENDPOINT", "http://garage.storage.svc:3903"
|
"GARAGE_ADMIN_ENDPOINT", "http://garage.storage.svc:3903"
|
||||||
)
|
)
|
||||||
|
HISTORY_FILE = "site-publish-history.json"
|
||||||
|
|
||||||
|
|
||||||
CACHE_CONTROL = "public, max-age=0, must-revalidate"
|
def validate_artifact_output(site_dir, artifact):
|
||||||
|
"""Prove every artifact and declared cache prefix exists before publishing any."""
|
||||||
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
|
if not html_dir.is_dir() or not any(path.is_file() for path in html_dir.rglob("*")):
|
||||||
|
die(f"artifact {artifact['name']} build output is absent or empty: {html_dir}")
|
||||||
|
for rule in artifact["cache_rules"]:
|
||||||
|
if not rule["path"]:
|
||||||
|
continue
|
||||||
|
cache_root = html_dir / rule["path"]
|
||||||
|
if not cache_root.exists() or not any(path.is_file() for path in cache_root.rglob("*")):
|
||||||
|
die(f"artifact {artifact['name']} cache path /{rule['path']} has no built files")
|
||||||
|
|
||||||
|
|
||||||
def s3_sync(site_name, site_dir, excludes=None):
|
def validate_publication_environment(cfg):
|
||||||
endpoint = os.environ.get("GARAGE_S3_ENDPOINT", DEFAULT_S3_ENDPOINT)
|
"""Resolve every declared credential before the first bucket is changed."""
|
||||||
html_dir = site_dir / "build" / "html"
|
for artifact in cfg["artifacts"]:
|
||||||
if not html_dir.exists():
|
env(artifact["credentials"]["access_key_env"])
|
||||||
die(f"build/html not found — did the build step run? ({html_dir})")
|
env(artifact["credentials"]["secret_key_env"])
|
||||||
env("AWS_ACCESS_KEY_ID")
|
if cfg["compatibility"] and cfg["aliases"] and not os.environ.get("GARAGE_ADMIN_TOKEN"):
|
||||||
env("AWS_SECRET_ACCESS_KEY")
|
die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
|
||||||
os.environ.setdefault("AWS_DEFAULT_REGION", "sjc001")
|
|
||||||
|
|
||||||
|
def _is_immutable(rule):
|
||||||
|
return "immutable" in {
|
||||||
|
part.strip().lower().split("=", 1)[0]
|
||||||
|
for part in rule["cache_control"].split(",")
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _aws_capture(args, aws_env):
|
||||||
|
"""Run a non-streaming AWS request without exposing environment credentials."""
|
||||||
|
print(f" $ {' '.join(str(part) for part in args)}")
|
||||||
|
return subprocess.run(args, env=aws_env, text=True, capture_output=True, check=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _immutable_head(endpoint, bucket, key, aws_env):
|
||||||
|
args = ["aws", "--endpoint-url", endpoint, "s3api", "head-object",
|
||||||
|
"--bucket", bucket, "--key", key, "--output", "json"]
|
||||||
|
result = _aws_capture(args, aws_env)
|
||||||
|
if result.returncode == 0:
|
||||||
|
return json.loads(result.stdout)
|
||||||
|
error = f"{result.stdout}\n{result.stderr}"
|
||||||
|
if any(marker in error for marker in ("404", "Not Found", "NoSuchKey")):
|
||||||
|
return None
|
||||||
|
raise RuntimeError(f"head-object failed for s3://{bucket}/{key}: {error.strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def _immutable_digests(source, cache_control, content_type):
|
||||||
|
with source.open("rb") as stream:
|
||||||
|
content_digest = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||||
|
publication = hashlib.sha256()
|
||||||
|
publication.update(cache_control.encode())
|
||||||
|
publication.update(b"\0")
|
||||||
|
publication.update(content_type.encode())
|
||||||
|
publication.update(b"\0")
|
||||||
|
with source.open("rb") as stream:
|
||||||
|
for block in iter(lambda: stream.read(1024 * 1024), b""):
|
||||||
|
publication.update(block)
|
||||||
|
return content_digest, publication.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _same_immutable_object(info, content_digest, publication_digest, cache_control, content_type):
|
||||||
|
metadata = {key.lower(): value for key, value in (info.get("Metadata") or {}).items()}
|
||||||
|
return (
|
||||||
|
metadata.get("sha256") == content_digest
|
||||||
|
and metadata.get("publication-sha256") == publication_digest
|
||||||
|
and info.get("CacheControl") == cache_control
|
||||||
|
and info.get("ContentType") == content_type
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def publish_immutable_file(endpoint, bucket, key, source, cache_control, aws_env):
|
||||||
|
"""Publish a content-addressed key; identical retries converge."""
|
||||||
|
content_type = mimetypes.guess_type(source.name)[0] or "application/octet-stream"
|
||||||
|
content_digest, publication_digest = _immutable_digests(source, cache_control, content_type)
|
||||||
|
address_digests = re.findall(r"(?<![0-9a-f])([0-9a-f]{64})(?![0-9a-f])", key.lower())
|
||||||
|
if address_digests != [publication_digest]:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"immutable key must contain its one publication SHA-256 {publication_digest}: "
|
||||||
|
f"s3://{bucket}/{key}"
|
||||||
|
)
|
||||||
|
existing = _immutable_head(endpoint, bucket, key, aws_env)
|
||||||
|
if existing is not None:
|
||||||
|
if _same_immutable_object(
|
||||||
|
existing, content_digest, publication_digest, cache_control, content_type,
|
||||||
|
):
|
||||||
|
print(f" Immutable object already matches: s3://{bucket}/{key}")
|
||||||
|
return False
|
||||||
|
raise RuntimeError(f"immutable object differs or lacks publisher digest: s3://{bucket}/{key}")
|
||||||
|
args = ["aws", "--endpoint-url", endpoint, "s3api", "put-object",
|
||||||
|
"--bucket", bucket, "--key", key, "--body", str(source),
|
||||||
|
"--content-type", content_type, "--cache-control", cache_control,
|
||||||
|
"--metadata", f"sha256={content_digest},publication-sha256={publication_digest}"]
|
||||||
|
result = _aws_capture(args, aws_env)
|
||||||
|
if result.returncode == 0:
|
||||||
|
return True
|
||||||
|
error = f"{result.stdout}\n{result.stderr}"
|
||||||
|
raise RuntimeError(f"put-object failed for s3://{bucket}/{key}: {error.strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def publish_immutable_rule(artifact, route, rule, html_dir, aws_env):
|
||||||
|
"""Publish one immutable cache partition without overwrite or deletion."""
|
||||||
|
rule_root = html_dir / rule["path"]
|
||||||
|
child_paths = [candidate["path"] for candidate in artifact["cache_rules"]
|
||||||
|
if candidate["path"].startswith(f"{rule['path'].rstrip('/')}/")]
|
||||||
|
object_prefix = route["path"].strip("/")
|
||||||
|
for source in sorted(path for path in rule_root.rglob("*") if path.is_file()):
|
||||||
|
relative = source.relative_to(html_dir).as_posix()
|
||||||
|
if any(relative == child or relative.startswith(f"{child}/") for child in child_paths):
|
||||||
|
continue
|
||||||
|
if any(fnmatch.fnmatch(relative, pattern) for pattern in artifact["excludes"]):
|
||||||
|
continue
|
||||||
|
key = "/".join(part for part in (object_prefix, relative) if part)
|
||||||
|
publish_immutable_file(
|
||||||
|
artifact["s3_endpoint"], artifact["bucket"], key, source,
|
||||||
|
rule["cache_control"], aws_env,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def publication_aws_env(artifact, credential_env_names=None):
|
||||||
|
"""Build the route-scoped AWS environment without leaking other credentials."""
|
||||||
|
access_key = env(artifact["credentials"]["access_key_env"])
|
||||||
|
secret_key = env(artifact["credentials"]["secret_key_env"])
|
||||||
|
aws_env = os.environ.copy()
|
||||||
|
for name in credential_env_names or artifact["credentials"].values():
|
||||||
|
aws_env.pop(name, None)
|
||||||
|
for name in ("CI_BOT_TOKEN", "GARAGE_ADMIN_TOKEN", "AWS_PROFILE",
|
||||||
|
"AWS_SHARED_CREDENTIALS_FILE", "AWS_SESSION_TOKEN"):
|
||||||
|
aws_env.pop(name, None)
|
||||||
|
aws_env.update({
|
||||||
|
"AWS_ACCESS_KEY_ID": access_key,
|
||||||
|
"AWS_SECRET_ACCESS_KEY": secret_key,
|
||||||
|
"AWS_DEFAULT_REGION": os.environ.get("AWS_DEFAULT_REGION", "sjc001"),
|
||||||
|
})
|
||||||
|
return aws_env
|
||||||
|
|
||||||
|
|
||||||
|
def publish_route_immutables(artifact, route, site_dir, credential_env_names=None):
|
||||||
|
"""Publish one route's immutable partitions during the global preflight."""
|
||||||
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
|
aws_env = publication_aws_env(artifact, credential_env_names)
|
||||||
|
for rule in artifact["cache_rules"]:
|
||||||
|
if _is_immutable(rule):
|
||||||
|
publish_immutable_rule(artifact, route, rule, html_dir, aws_env)
|
||||||
|
|
||||||
|
|
||||||
|
def retired_immutable_filters(artifact, route, html_dir, previous_contract):
|
||||||
|
"""Protect every historical immutable prefix inside the current sync scope."""
|
||||||
|
if not previous_contract:
|
||||||
|
return []
|
||||||
|
current_prefix = route["path"].strip("/")
|
||||||
|
filters = []
|
||||||
|
current_immutable = set(immutable_prefixes(artifact, route))
|
||||||
|
for immutable_prefix in previous_contract["immutable_prefixes"]:
|
||||||
|
if current_prefix:
|
||||||
|
marker = f"{current_prefix}/"
|
||||||
|
if not immutable_prefix.startswith(marker):
|
||||||
|
continue
|
||||||
|
relative_path = immutable_prefix[len(marker):]
|
||||||
|
else:
|
||||||
|
relative_path = immutable_prefix
|
||||||
|
collision = html_dir / relative_path
|
||||||
|
if (immutable_prefix not in current_immutable and collision.exists()
|
||||||
|
and any(path.is_file() for path in collision.rglob("*"))):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"current artifact collides with retired immutable partition: {relative_path}"
|
||||||
|
)
|
||||||
|
filters.extend(("--exclude", f"{relative_path}/*"))
|
||||||
|
return filters
|
||||||
|
|
||||||
|
|
||||||
|
def immutable_prefixes(artifact, route):
|
||||||
|
route_prefix = route["path"].strip("/")
|
||||||
|
return sorted({
|
||||||
|
"/".join(part for part in (route_prefix, rule["path"]) if part)
|
||||||
|
for rule in artifact["cache_rules"] if _is_immutable(rule)
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_contract=None):
|
||||||
|
endpoint = artifact["s3_endpoint"]
|
||||||
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
|
aws_env = publication_aws_env(artifact, credential_env_names)
|
||||||
|
bucket = artifact["bucket"]
|
||||||
|
object_prefix = route["path"].strip("/")
|
||||||
|
destination = f"s3://{bucket}/{object_prefix + '/' if object_prefix else ''}"
|
||||||
|
default_cache = next(rule["cache_control"] for rule in artifact["cache_rules"] if not rule["path"])
|
||||||
|
immutable_paths = [rule["path"] for rule in artifact["cache_rules"] if _is_immutable(rule)]
|
||||||
# `excludes` are patterns (site.yaml `excludes:` list) that should never
|
# `excludes` are patterns (site.yaml `excludes:` list) that should never
|
||||||
# be uploaded *and* should never be deleted from the bucket — escape hatch
|
# be uploaded *and* should never be deleted from the bucket — escape hatch
|
||||||
# for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli).
|
# for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli).
|
||||||
exclude_flags = " ".join(f"--exclude {shlex.quote(p)}" for p in (excludes or []))
|
exclude_args = [arg for pattern in artifact["excludes"] for arg in ("--exclude", pattern)]
|
||||||
if excludes:
|
if artifact["excludes"]:
|
||||||
print(f"Excluding patterns: {excludes}")
|
print(f"Excluding patterns: {artifact['excludes']}")
|
||||||
print(f"Syncing {html_dir} → s3://{site_name} via {endpoint}")
|
print(f"Syncing artifact {artifact['name']} → {destination} via {endpoint}")
|
||||||
# `sync --delete` handles new/changed/orphaned files. `cp --recursive`
|
# Upload with the final cache policy before cleanup. Sync and deletion are
|
||||||
# then re-uploads everything to refresh metadata (cache-control,
|
# scoped to the same current route prefix and cache partition. A route move
|
||||||
# content-type) on objects sync skipped because nothing changed.
|
# leaves its old bucket partition intact but unreachable after the old
|
||||||
# Cost: a no-op deploy still re-uploads every byte. Sites here are
|
# Ingress disappears, while stale mutable keys on the serving prefix are
|
||||||
# small enough that that's free; correctness wins over throughput.
|
# deleted. Immutable subtrees are structurally excluded. `cp --recursive`
|
||||||
|
# refreshes metadata atomically per object before `sync --delete` removes
|
||||||
|
# stale keys without ever exposing new bytes under a provisional policy.
|
||||||
|
# A no-op deploy therefore transfers the artifact bytes once.
|
||||||
# AWS CLI guesses Content-Type from file extension on local→S3 uploads,
|
# AWS CLI guesses Content-Type from file extension on local→S3 uploads,
|
||||||
# so a fresh upload always carries the right MIME type.
|
# so a fresh upload always carries the right MIME type.
|
||||||
run(
|
specific_paths = [rule["path"] for rule in artifact["cache_rules"] if rule["path"]]
|
||||||
f"aws --endpoint-url {endpoint} s3 sync {html_dir}/ s3://{site_name}/ "
|
default_filters = [arg for path in specific_paths for arg in ("--exclude", f"{path}/*")]
|
||||||
f"--delete --only-show-errors "
|
retired_filters = retired_immutable_filters(artifact, route, html_dir, previous_contract)
|
||||||
f"--cache-control '{CACHE_CONTROL}' "
|
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
||||||
f"{exclude_flags}".rstrip()
|
"--recursive", "--only-show-errors", "--cache-control", default_cache,
|
||||||
)
|
*default_filters, *retired_filters, *exclude_args], env=aws_env)
|
||||||
print("Re-stamping metadata on all objects...")
|
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
|
||||||
run(
|
"--delete", "--only-show-errors", "--cache-control", default_cache,
|
||||||
f"aws --endpoint-url {endpoint} s3 cp {html_dir}/ s3://{site_name}/ "
|
*default_filters, *retired_filters, *exclude_args], env=aws_env)
|
||||||
f"--recursive --only-show-errors "
|
for rule in artifact["cache_rules"]:
|
||||||
f"--cache-control '{CACHE_CONTROL}' "
|
if not rule["path"]:
|
||||||
f"{exclude_flags}".rstrip()
|
continue
|
||||||
)
|
if _is_immutable(rule):
|
||||||
|
continue
|
||||||
|
include = f"{rule['path'].rstrip('/')}/*"
|
||||||
|
child_filters = [arg for path in specific_paths
|
||||||
|
if path.startswith(f"{rule['path'].rstrip('/')}/")
|
||||||
|
for arg in ("--exclude", f"{path}/*")]
|
||||||
|
# Apply rules from the artifact root so artifact-level exclusions keep
|
||||||
|
# their original meaning under every cache override.
|
||||||
|
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
||||||
|
"--recursive", "--only-show-errors", "--cache-control", rule["cache_control"],
|
||||||
|
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
|
||||||
|
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
|
||||||
|
"--delete", "--only-show-errors", "--cache-control", rule["cache_control"],
|
||||||
|
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
|
||||||
|
|
||||||
|
|
||||||
def garage_admin(method, path, token, body=None):
|
def garage_admin(method, path, token, body=None):
|
||||||
@@ -89,15 +285,13 @@ def ensure_bucket_aliases(site_name, aliases, admin_token):
|
|||||||
if not aliases:
|
if not aliases:
|
||||||
return
|
return
|
||||||
if not admin_token:
|
if not admin_token:
|
||||||
print(" (no GARAGE_ADMIN_TOKEN — skipping bucket alias reconcile)")
|
die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
info = garage_admin("GET", f"/v2/GetBucketInfo?globalAlias={site_name}",
|
info = garage_admin("GET", f"/v2/GetBucketInfo?globalAlias={site_name}",
|
||||||
admin_token)
|
admin_token)
|
||||||
except (HTTPError, URLError) as e:
|
except (HTTPError, URLError) as e:
|
||||||
print(f" WARNING: bucket lookup failed: {e}")
|
raise RuntimeError(f"bucket lookup failed for {site_name}: {e}") from e
|
||||||
return
|
|
||||||
|
|
||||||
bucket_id = info.get("id")
|
bucket_id = info.get("id")
|
||||||
existing = set(info.get("globalAliases") or [])
|
existing = set(info.get("globalAliases") or [])
|
||||||
@@ -120,46 +314,168 @@ def render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg):
|
|||||||
"""Always re-render manifests from current site.yaml. Templates own
|
"""Always re-render manifests from current site.yaml. Templates own
|
||||||
domain + aliases, so changes propagate without manual edits."""
|
domain + aliases, so changes propagate without manual edits."""
|
||||||
manifests_dir.mkdir(parents=True, exist_ok=True)
|
manifests_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
routes = []
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
artifact = artifact_by_name[route["artifact"]]
|
||||||
|
resource_name = k8s_name(site_name) if cfg["compatibility"] else f"{k8s_name(site_name)}-{route['name']}"
|
||||||
|
routes.append({**route, "resource_name": resource_name, "artifact_config": artifact})
|
||||||
template_vars = {
|
template_vars = {
|
||||||
"site": site_name,
|
"site": site_name,
|
||||||
"site_k8s": k8s_name(site_name),
|
"site_k8s": k8s_name(site_name),
|
||||||
"domain": cfg["domain"],
|
"domain": cfg["domain"],
|
||||||
"aliases": cfg["aliases"],
|
"aliases": cfg["aliases"],
|
||||||
"namespace": NAMESPACE,
|
"namespace": NAMESPACE,
|
||||||
"middlewares": cfg["middlewares"],
|
"compatibility": cfg["compatibility"],
|
||||||
|
"routes": routes,
|
||||||
}
|
}
|
||||||
render_templates(action_dir, template_vars, app_dir, manifests_dir)
|
render_templates(action_dir, template_vars, app_dir, manifests_dir)
|
||||||
|
|
||||||
|
|
||||||
def deploy_static(site_name, site_dir, action_dir, token, cfg):
|
def previous_route_contracts(app_dir):
|
||||||
s3_sync(site_name, site_dir, excludes=cfg.get("excludes"))
|
"""Read the append-only bucket history kept beside generated manifests."""
|
||||||
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
|
path = app_dir / HISTORY_FILE
|
||||||
|
if not path.exists():
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
document = json.loads(path.read_text())
|
||||||
|
except (OSError, json.JSONDecodeError) as exc:
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}") from exc
|
||||||
|
if (not isinstance(document, dict) or set(document) != {"schemaVersion", "buckets"}
|
||||||
|
or document["schemaVersion"] != 1 or not isinstance(document["buckets"], dict)):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}")
|
||||||
|
for bucket, contract in document["buckets"].items():
|
||||||
|
if (not isinstance(bucket, str) or not isinstance(contract, dict)
|
||||||
|
or set(contract) != {"access", "artifact", "immutablePrefixes", "routePath"}
|
||||||
|
or contract["access"] not in {"legacy", "protected", "public"}
|
||||||
|
or not isinstance(contract["artifact"], str)
|
||||||
|
or not isinstance(contract["routePath"], str)
|
||||||
|
or not contract["routePath"].startswith("/")
|
||||||
|
or not isinstance(contract["immutablePrefixes"], list)
|
||||||
|
or any(not _valid_immutable_prefix(value)
|
||||||
|
for value in contract["immutablePrefixes"])
|
||||||
|
or len(contract["immutablePrefixes"]) != len(set(contract["immutablePrefixes"]))):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}")
|
||||||
|
return {
|
||||||
|
bucket: {
|
||||||
|
"access": contract["access"],
|
||||||
|
"artifact": contract["artifact"],
|
||||||
|
"immutable_prefixes": sorted(contract["immutablePrefixes"]),
|
||||||
|
"path": contract["routePath"],
|
||||||
|
}
|
||||||
|
for bucket, contract in document["buckets"].items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def next_route_contracts(cfg, previous_contracts):
|
||||||
|
"""Carry protected access and immutable prefixes forward for every known bucket."""
|
||||||
|
contracts = json.loads(json.dumps(previous_contracts))
|
||||||
|
artifacts = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
artifact = artifacts[route["artifact"]]
|
||||||
|
bucket = artifact["bucket"]
|
||||||
|
previous = previous_contracts.get(bucket)
|
||||||
|
access = "protected" if (
|
||||||
|
route["access"] == "protected" or previous and previous["access"] == "protected"
|
||||||
|
) else route["access"]
|
||||||
|
contracts[bucket] = {
|
||||||
|
"access": access,
|
||||||
|
"artifact": route["artifact"],
|
||||||
|
"immutable_prefixes": sorted(set(
|
||||||
|
(previous or {}).get("immutable_prefixes", []) + immutable_prefixes(artifact, route)
|
||||||
|
)),
|
||||||
|
"path": route["path"],
|
||||||
|
}
|
||||||
|
return contracts
|
||||||
|
|
||||||
|
|
||||||
|
def write_route_contracts(app_dir, contracts):
|
||||||
|
app_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
document = {
|
||||||
|
"schemaVersion": 1,
|
||||||
|
"buckets": {
|
||||||
|
bucket: {
|
||||||
|
"access": contract["access"],
|
||||||
|
"artifact": contract["artifact"],
|
||||||
|
"immutablePrefixes": contract["immutable_prefixes"],
|
||||||
|
"routePath": contract["path"],
|
||||||
|
}
|
||||||
|
for bucket, contract in sorted(contracts.items())
|
||||||
|
},
|
||||||
|
}
|
||||||
|
(app_dir / HISTORY_FILE).write_text(f"{json.dumps(document, indent=2, sort_keys=True)}\n")
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_immutable_prefix(value):
|
||||||
|
return (isinstance(value, str) and value and not value.startswith("/")
|
||||||
|
and not value.endswith("/") and ".." not in PurePosixPath(value).parts)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_route_migrations(cfg, previous_contracts):
|
||||||
|
artifacts = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
artifact = artifacts[route["artifact"]]
|
||||||
|
previous = previous_contracts.get(artifact["bucket"])
|
||||||
|
if (previous and previous["access"] == "protected"
|
||||||
|
and route["access"] in {"public", "legacy"}
|
||||||
|
):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"artifact {route['artifact']} cannot become public while reusing protected "
|
||||||
|
f"bucket {artifact['bucket']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def deploy_static(site_name, site_dir, action_dir, token, cfg):
|
||||||
|
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
credential_env_names = {
|
||||||
|
name for artifact in cfg["artifacts"] for name in artifact["credentials"].values()
|
||||||
|
}
|
||||||
|
validate_publication_environment(cfg)
|
||||||
|
for artifact in cfg["artifacts"]:
|
||||||
|
validate_artifact_output(site_dir, artifact)
|
||||||
apps_dir = clone_apps(token)
|
apps_dir = clone_apps(token)
|
||||||
app_dir = apps_dir / "sjc001" / "websites" / site_name
|
app_dir = apps_dir / "sjc001" / "websites" / site_name
|
||||||
manifests_dir = app_dir / "manifests"
|
manifests_dir = app_dir / "manifests"
|
||||||
|
previous_contracts = previous_route_contracts(app_dir)
|
||||||
|
validate_route_migrations(cfg, previous_contracts)
|
||||||
|
# Complete immutable work across the whole publication before any route's
|
||||||
|
# mutable pointers can change. Partial immutable success is safe; mixing a
|
||||||
|
# new route with an old route after a later immutable failure is not.
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
publish_route_immutables(
|
||||||
|
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
|
||||||
|
)
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
s3_sync(
|
||||||
|
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
|
||||||
|
previous_contracts.get(artifact_by_name[route["artifact"]]["bucket"]),
|
||||||
|
)
|
||||||
|
if cfg["compatibility"]:
|
||||||
|
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
|
||||||
|
|
||||||
|
write_route_contracts(app_dir, next_route_contracts(cfg, previous_contracts))
|
||||||
render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg)
|
render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg)
|
||||||
|
|
||||||
commit_and_push(apps_dir, f"Deploy {site_name}")
|
commit_and_push(apps_dir, f"Deploy {site_name}", token)
|
||||||
|
|
||||||
|
|
||||||
def decommission(site_name, token):
|
def decommission(site_name, token, buckets=None):
|
||||||
"""Remove manifests from apps repo."""
|
"""Remove manifests from apps repo."""
|
||||||
user = env("CI_BOT_USER", "ci-bot")
|
apps_dir = clone_apps(token)
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
|
||||||
apps_dir = Path(tmp)
|
|
||||||
run(f"git clone --depth 1 https://{user}:{token}@{GITEA_HOST}/fritzlab/apps.git {apps_dir}")
|
|
||||||
site_path = apps_dir / "sjc001" / "websites" / site_name
|
site_path = apps_dir / "sjc001" / "websites" / site_name
|
||||||
if not site_path.exists():
|
if not site_path.exists():
|
||||||
print(f"No manifests for {site_name} — nothing to remove")
|
print(f"No manifests for {site_name} — nothing to remove")
|
||||||
return
|
return
|
||||||
|
history_path = site_path / HISTORY_FILE
|
||||||
|
history = history_path.read_bytes() if history_path.exists() else None
|
||||||
shutil.rmtree(site_path)
|
shutil.rmtree(site_path)
|
||||||
run(f"git -C {apps_dir} config user.name {user}")
|
if history is not None:
|
||||||
run(f"git -C {apps_dir} config user.email {user}@fritzlab.net")
|
site_path.mkdir(parents=True)
|
||||||
commit_and_push(apps_dir, f"Decommission {site_name}")
|
(site_path / HISTORY_FILE).write_bytes(history)
|
||||||
print(f"Bucket {site_name} and its objects are NOT purged automatically.")
|
commit_and_push(apps_dir, f"Decommission {site_name}", token)
|
||||||
print(f" garage bucket delete {site_name} --yes")
|
for bucket in buckets or [site_name]:
|
||||||
|
print(f"Bucket {bucket} and its objects are NOT purged automatically.")
|
||||||
|
print(f" garage bucket delete {bucket} --yes")
|
||||||
|
|
||||||
|
|
||||||
def cmd_deploy():
|
def cmd_deploy():
|
||||||
@@ -173,7 +489,8 @@ def cmd_deploy():
|
|||||||
|
|
||||||
if not cfg["enabled"]:
|
if not cfg["enabled"]:
|
||||||
print("Site disabled — running decommission...")
|
print("Site disabled — running decommission...")
|
||||||
decommission(site_name, token)
|
decommission(site_name, token, [artifact["bucket"] for artifact in cfg["artifacts"]])
|
||||||
return
|
return
|
||||||
|
|
||||||
|
validate_artifact_inputs(site_dir, cfg)
|
||||||
deploy_static(site_name, site_dir, action_dir, token, cfg)
|
deploy_static(site_name, site_dir, action_dir, token, cfg)
|
||||||
|
|||||||
Executable
+5
@@ -0,0 +1,5 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
case "$1" in
|
||||||
|
*Username*) printf '%s\n' "$CI_BOT_USER" ;;
|
||||||
|
*) printf '%s\n' "$CI_BOT_TOKEN" ;;
|
||||||
|
esac
|
||||||
+3
-2
@@ -24,9 +24,10 @@ def ensure_aws():
|
|||||||
subprocess.run(["aws", "--version"], check=True)
|
subprocess.run(["aws", "--version"], check=True)
|
||||||
|
|
||||||
|
|
||||||
def ensure_jinja2():
|
def ensure_python_dependencies():
|
||||||
try:
|
try:
|
||||||
import jinja2
|
import jinja2
|
||||||
|
import yaml
|
||||||
except ImportError:
|
except ImportError:
|
||||||
print("Installing jinja2 + pyyaml...")
|
print("Installing jinja2 + pyyaml...")
|
||||||
subprocess.run(
|
subprocess.run(
|
||||||
@@ -36,6 +37,6 @@ def ensure_jinja2():
|
|||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
ensure_jinja2()
|
ensure_python_dependencies()
|
||||||
ensure_aws()
|
ensure_aws()
|
||||||
print("Setup complete")
|
print("Setup complete")
|
||||||
|
|||||||
+462
-69
@@ -1,26 +1,34 @@
|
|||||||
"""Shared utilities for the site-publish action."""
|
"""Shared utilities for the site-publish action."""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path, PurePosixPath
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
from jinja2 import Environment, FileSystemLoader
|
from jinja2 import Environment, FileSystemLoader, StrictUndefined
|
||||||
|
|
||||||
APPS_REPO = "fritzlab/apps"
|
APPS_REPO = "fritzlab/apps"
|
||||||
GITEA_HOST = "code.fritzlab.net"
|
GITEA_HOST = "code.fritzlab.net"
|
||||||
NAMESPACE = "websites"
|
NAMESPACE = "websites"
|
||||||
DEFAULT_S3_ENDPOINT = "http://garage-s3.storage.svc:3900"
|
DEFAULT_S3_ENDPOINT = "http://garage-s3.storage.svc:3900"
|
||||||
|
DEFAULT_WEBSITE_SUFFIX = "web.sjc001.fritzlab.net"
|
||||||
|
DEFAULT_CACHE_CONTROL = "public, max-age=0, must-revalidate"
|
||||||
|
|
||||||
EXCLUDE_FILES = {
|
EXCLUDE_FILES = {
|
||||||
".git", ".gitea", ".gitignore", "site.yaml",
|
".git", ".gitea", ".gitignore", "site.yaml", "build", ".site-publish",
|
||||||
"build", "Makefile", "README.md", "CLAUDE.md",
|
"Makefile", "README.md", "CLAUDE.md", "Dockerfile", ".dockerignore",
|
||||||
"Dockerfile", ".dockerignore", "go.mod", "go.sum",
|
"go.mod", "go.sum",
|
||||||
}
|
}
|
||||||
|
|
||||||
VALID_TYPES = {"static", "hugo", "mkdocs"}
|
VALID_TYPES = {"static", "hugo", "mkdocs"}
|
||||||
|
NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")
|
||||||
|
ENV_RE = re.compile(r"^[A-Z_][A-Z0-9_]*$")
|
||||||
|
ACCESS_KEY_ENV_RE = re.compile(r"^([A-Z][A-Z0-9_]*)_S3_ACCESS_KEY$")
|
||||||
|
BUCKET_RE = re.compile(r"^[a-z0-9](?:[a-z0-9.-]{1,61}[a-z0-9])?$")
|
||||||
|
MIDDLEWARE_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9])?$")
|
||||||
|
|
||||||
DOCKER_DEPRECATION_MSG = """\
|
DOCKER_DEPRECATION_MSG = """\
|
||||||
type: docker is no longer supported by action/site-publish.
|
type: docker is no longer supported by action/site-publish.
|
||||||
@@ -42,113 +50,498 @@ example.\
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigError(ValueError):
|
||||||
|
"""A site.yaml contract violation."""
|
||||||
|
|
||||||
|
|
||||||
def k8s_name(name):
|
def k8s_name(name):
|
||||||
"""Sanitize for DNS-1035 label (dots → dashes)."""
|
|
||||||
return name.replace(".", "-")
|
return name.replace(".", "-")
|
||||||
|
|
||||||
|
|
||||||
def env(key, default=None):
|
def env(key, default=None):
|
||||||
val = os.environ.get(key, default)
|
val = os.environ.get(key, default)
|
||||||
if val is None:
|
if val is None or val == "":
|
||||||
die(f"Missing required env var: {key}")
|
die(f"Missing required env var: {key}")
|
||||||
return val
|
return val
|
||||||
|
|
||||||
|
|
||||||
def die(msg):
|
def die(msg):
|
||||||
print(f"ERROR: {msg}", file=sys.stderr)
|
print(f"ERROR: {msg}", file=sys.stderr)
|
||||||
sys.exit(1)
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
def run(cmd, **kwargs):
|
def run(cmd, *, display=None, **kwargs):
|
||||||
print(f" $ {cmd}")
|
"""Run an argv command, printing only a safe display form."""
|
||||||
return subprocess.run(cmd, shell=True, check=True, **kwargs)
|
if not isinstance(cmd, (list, tuple)):
|
||||||
|
raise TypeError("run() requires an argv list")
|
||||||
|
shown = display if display is not None else " ".join(str(part) for part in cmd)
|
||||||
|
print(f" $ {shown}")
|
||||||
|
return subprocess.run(cmd, check=True, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
def parse_site_yaml(site_dir):
|
def _mapping(value, label):
|
||||||
path = Path(site_dir) / "site.yaml"
|
if not isinstance(value, dict):
|
||||||
if not path.exists():
|
raise ConfigError(f"{label} must be a mapping")
|
||||||
die("site.yaml not found in repo root")
|
return value
|
||||||
|
|
||||||
with open(path) as f:
|
|
||||||
cfg = yaml.safe_load(f)
|
|
||||||
|
|
||||||
if not cfg.get("domain"):
|
def _list(value, label):
|
||||||
die("domain is required in site.yaml")
|
if not isinstance(value, list):
|
||||||
|
raise ConfigError(f"{label} must be a list")
|
||||||
|
return value
|
||||||
|
|
||||||
site_type = cfg.get("type", "static")
|
|
||||||
|
|
||||||
|
def _known_keys(value, allowed, label):
|
||||||
|
unknown = set(value) - set(allowed)
|
||||||
|
if unknown:
|
||||||
|
raise ConfigError(f"{label} has unknown fields: {', '.join(sorted(unknown))}")
|
||||||
|
|
||||||
|
|
||||||
|
def _strings(value, label):
|
||||||
|
values = _list(value or [], label)
|
||||||
|
if any(not isinstance(item, str) or not item for item in values):
|
||||||
|
raise ConfigError(f"{label} must be a list of non-empty strings")
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def _hostname(value, label):
|
||||||
|
if not isinstance(value, str) or len(value) > 253 or value.endswith("."):
|
||||||
|
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
|
||||||
|
labels = value.split(".")
|
||||||
|
if len(labels) < 2 or any(not NAME_RE.fullmatch(part) for part in labels):
|
||||||
|
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _aliases(value, domain):
|
||||||
|
aliases = _strings(value or [], "aliases")
|
||||||
|
aliases = [_hostname(alias, "aliases entry") for alias in aliases]
|
||||||
|
if len(aliases) != len(set(aliases)) or domain in aliases:
|
||||||
|
raise ConfigError("aliases must be unique and cannot repeat domain")
|
||||||
|
return aliases
|
||||||
|
|
||||||
|
|
||||||
|
def _relative_path(value, label, *, allow_root=False):
|
||||||
|
value = "" if value is None else value
|
||||||
|
if not isinstance(value, str):
|
||||||
|
raise ConfigError(f"{label} must be a string")
|
||||||
|
if value == "/" and allow_root:
|
||||||
|
return ""
|
||||||
|
path = PurePosixPath(value)
|
||||||
|
if path.is_absolute() or ".." in path.parts:
|
||||||
|
raise ConfigError(f"{label} must be a relative path without '..'")
|
||||||
|
normalized = str(path).strip("/")
|
||||||
|
return "" if normalized == "." else normalized
|
||||||
|
|
||||||
|
|
||||||
|
def _route_path(value, label):
|
||||||
|
if not isinstance(value, str) or not value.startswith("/"):
|
||||||
|
raise ConfigError(f"{label} must start with '/'")
|
||||||
|
if "//" in value or "?" in value or "#" in value or ".." in value.split("/"):
|
||||||
|
raise ConfigError(f"{label} is not a canonical URL path")
|
||||||
|
normalized = value.rstrip("/") or "/"
|
||||||
|
if not re.fullmatch(r"/[A-Za-z0-9._~/-]*", normalized):
|
||||||
|
raise ConfigError(f"{label} contains unsupported URL path characters")
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def _middlewares(value, label):
|
||||||
|
names = _strings(value, label)
|
||||||
|
if any(not MIDDLEWARE_RE.fullmatch(name) for name in names):
|
||||||
|
raise ConfigError(f"{label} contains an invalid middleware name")
|
||||||
|
if len(names) != len(set(names)):
|
||||||
|
raise ConfigError(f"{label} contains duplicate middleware names")
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def _cache_control(value, label):
|
||||||
|
if not isinstance(value, str) or not value.strip():
|
||||||
|
raise ConfigError(f"{label} must be a non-empty Cache-Control value")
|
||||||
|
directives = [part.strip().lower() for part in value.split(",")]
|
||||||
|
names = [part.split("=", 1)[0] for part in directives]
|
||||||
|
if len(names) != len(set(names)):
|
||||||
|
raise ConfigError(f"{label} repeats a Cache-Control directive")
|
||||||
|
present = set(names)
|
||||||
|
if {"public", "private"} <= present:
|
||||||
|
raise ConfigError(f"{label} cannot be both public and private")
|
||||||
|
ages = {}
|
||||||
|
for part in directives:
|
||||||
|
name = part.split("=", 1)[0]
|
||||||
|
if name in {"max-age", "s-maxage"}:
|
||||||
|
raw = part.split("=", 1)[1] if "=" in part else ""
|
||||||
|
if not raw.isdigit():
|
||||||
|
raise ConfigError(f"{label} {name} must be a non-negative integer")
|
||||||
|
ages[name] = int(raw)
|
||||||
|
max_age = ages.get("max-age")
|
||||||
|
if "immutable" in present and (not max_age or present & {"no-store", "no-cache", "must-revalidate"}):
|
||||||
|
raise ConfigError(f"{label} immutable requires positive max-age without revalidation")
|
||||||
|
if "no-store" in present and any(ages.values()):
|
||||||
|
raise ConfigError(f"{label} no-store contradicts positive caching")
|
||||||
|
if "private" in present and ages.get("s-maxage"):
|
||||||
|
raise ConfigError(f"{label} private contradicts shared-cache max-age")
|
||||||
|
return ", ".join(part.strip() for part in value.split(","))
|
||||||
|
|
||||||
|
|
||||||
|
def _site_type(value, label):
|
||||||
|
site_type = value or "static"
|
||||||
if site_type == "docker":
|
if site_type == "docker":
|
||||||
die(DOCKER_DEPRECATION_MSG)
|
raise ConfigError(DOCKER_DEPRECATION_MSG)
|
||||||
|
|
||||||
if site_type not in VALID_TYPES:
|
if site_type not in VALID_TYPES:
|
||||||
die(f"Unknown site type: {site_type} (valid: {', '.join(sorted(VALID_TYPES))})")
|
raise ConfigError(f"Unknown {label}: {site_type} (valid: {', '.join(sorted(VALID_TYPES))})")
|
||||||
|
return site_type
|
||||||
|
|
||||||
excludes = cfg.get("excludes") or []
|
|
||||||
if not isinstance(excludes, list) or any(not isinstance(p, str) for p in excludes):
|
|
||||||
die("excludes must be a list of string patterns")
|
|
||||||
|
|
||||||
middlewares = cfg.get("middlewares") or []
|
def _endpoint(value, label):
|
||||||
if not isinstance(middlewares, list) or any(not isinstance(m, str) for m in middlewares):
|
parsed = urlparse(value)
|
||||||
die("middlewares must be a list of Traefik file-provider middleware names")
|
if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.path not in {"", "/"}:
|
||||||
|
raise ConfigError(f"{label} must be an http(s) origin without a path")
|
||||||
|
return value.rstrip("/")
|
||||||
|
|
||||||
site = {
|
|
||||||
"domain": cfg["domain"],
|
def _legacy_config(raw, site_name):
|
||||||
"type": site_type,
|
if not isinstance(raw.get("tidy", True), bool):
|
||||||
"enabled": cfg.get("enabled", True),
|
raise ConfigError("tidy must be a boolean")
|
||||||
"aliases": cfg.get("aliases") or [],
|
if not isinstance(raw.get("enabled", True), bool):
|
||||||
"content_dir": cfg.get("content_dir", ""),
|
raise ConfigError("enabled must be a boolean")
|
||||||
"tidy": cfg.get("tidy", True),
|
artifact = {
|
||||||
"excludes": excludes,
|
"name": "site",
|
||||||
|
"type": _site_type(raw.get("type", "static"), "site type"),
|
||||||
|
"content_dir": _relative_path(raw.get("content_dir", ""), "content_dir"),
|
||||||
|
"tidy": raw.get("tidy", True),
|
||||||
|
"excludes": _strings(raw.get("excludes") or [], "excludes"),
|
||||||
|
"build_dir": "build/html",
|
||||||
|
"bucket": site_name,
|
||||||
|
"s3_endpoint": os.environ.get("GARAGE_S3_ENDPOINT") or DEFAULT_S3_ENDPOINT,
|
||||||
|
"website_authority": "garage-s3.storage.svc.k8s.sjc001.fritzlab.net",
|
||||||
|
"credentials": {"access_key_env": "AWS_ACCESS_KEY_ID", "secret_key_env": "AWS_SECRET_ACCESS_KEY"},
|
||||||
|
"cache_rules": [{"path": "", "cache_control": DEFAULT_CACHE_CONTROL}],
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"version": 1,
|
||||||
|
"compatibility": "single-surface-v1",
|
||||||
|
"domain": raw["domain"],
|
||||||
|
"aliases": _aliases(raw.get("aliases"), raw["domain"]),
|
||||||
|
"enabled": raw.get("enabled", True),
|
||||||
|
"artifacts": [artifact],
|
||||||
|
"routes": [{
|
||||||
|
"name": "site", "path": "/", "artifact": "site", "access": "legacy",
|
||||||
|
"access_middleware": None,
|
||||||
|
"middlewares": _middlewares(raw.get("middlewares") or [], "middlewares"),
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _artifact(item, index):
|
||||||
|
label = f"artifacts[{index}]"
|
||||||
|
item = _mapping(item, label)
|
||||||
|
_known_keys(item, {"name", "type", "content_dir", "tidy", "excludes", "publish", "cache"}, label)
|
||||||
|
name = item.get("name")
|
||||||
|
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
|
||||||
|
raise ConfigError(f"{label}.name must be a DNS label")
|
||||||
|
publish = _mapping(item.get("publish"), f"{label}.publish")
|
||||||
|
_known_keys(publish, {"bucket", "credentials"}, f"{label}.publish")
|
||||||
|
bucket = publish.get("bucket")
|
||||||
|
if not isinstance(bucket, str) or not BUCKET_RE.fullmatch(bucket):
|
||||||
|
raise ConfigError(f"{label}.publish.bucket is not a valid bucket name")
|
||||||
|
credentials = _mapping(publish.get("credentials"), f"{label}.publish.credentials")
|
||||||
|
_known_keys(credentials, {"access_key_env", "secret_key_env"}, f"{label}.publish.credentials")
|
||||||
|
normalized_credentials = {}
|
||||||
|
for key in ("access_key_env", "secret_key_env"):
|
||||||
|
value = credentials.get(key)
|
||||||
|
if not isinstance(value, str) or not ENV_RE.fullmatch(value):
|
||||||
|
raise ConfigError(f"{label}.publish.credentials.{key} must name an environment variable")
|
||||||
|
normalized_credentials[key] = value
|
||||||
|
access_match = ACCESS_KEY_ENV_RE.fullmatch(normalized_credentials["access_key_env"])
|
||||||
|
expected_secret = (
|
||||||
|
f"{access_match.group(1)}_S3_SECRET_KEY" if access_match else None
|
||||||
|
)
|
||||||
|
if normalized_credentials["secret_key_env"] != expected_secret:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{label}.publish.credentials must be a matched "
|
||||||
|
"<NAME>_S3_ACCESS_KEY and <NAME>_S3_SECRET_KEY pair"
|
||||||
|
)
|
||||||
|
cache = _mapping(item.get("cache"), f"{label}.cache")
|
||||||
|
_known_keys(cache, {"rules"}, f"{label}.cache")
|
||||||
|
rules = _list(cache.get("rules"), f"{label}.cache.rules")
|
||||||
|
if not rules:
|
||||||
|
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
|
||||||
|
cache_rules, paths = [], set()
|
||||||
|
for rule_index, rule in enumerate(rules):
|
||||||
|
rule_label = f"{label}.cache.rules[{rule_index}]"
|
||||||
|
rule = _mapping(rule, rule_label)
|
||||||
|
_known_keys(rule, {"path", "cache_control"}, rule_label)
|
||||||
|
path = _relative_path(rule.get("path"), f"{rule_label}.path", allow_root=True)
|
||||||
|
if path in paths:
|
||||||
|
raise ConfigError(f"{label}.cache.rules has duplicate path /{path}")
|
||||||
|
paths.add(path)
|
||||||
|
cache_rules.append({"path": path, "cache_control": _cache_control(
|
||||||
|
rule.get("cache_control"), f"{rule_label}.cache_control"
|
||||||
|
)})
|
||||||
|
if "" not in paths:
|
||||||
|
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
|
||||||
|
cache_rules.sort(key=lambda rule: (len(PurePosixPath(rule["path"]).parts), rule["path"]))
|
||||||
|
immutable_paths = [rule["path"] for rule in cache_rules
|
||||||
|
if "immutable" in {part.strip().lower().split("=", 1)[0]
|
||||||
|
for part in rule["cache_control"].split(",")}]
|
||||||
|
if "" in immutable_paths:
|
||||||
|
raise ConfigError(f"{label}.cache.rules immutable paths must be narrower than '/'")
|
||||||
|
for path in immutable_paths:
|
||||||
|
if any(other.startswith(f"{path}/") for other in paths):
|
||||||
|
raise ConfigError(f"{label}.cache.rules cannot nest another policy under immutable /{path}")
|
||||||
|
authority = f"{bucket}.{DEFAULT_WEBSITE_SUFFIX}"
|
||||||
|
if not isinstance(item.get("tidy", True), bool):
|
||||||
|
raise ConfigError(f"{label}.tidy must be a boolean")
|
||||||
|
return {
|
||||||
|
"name": name,
|
||||||
|
"type": _site_type(item.get("type", "static"), f"{label}.type"),
|
||||||
|
"content_dir": _relative_path(item.get("content_dir", ""), f"{label}.content_dir"),
|
||||||
|
"tidy": item.get("tidy", True),
|
||||||
|
"excludes": _strings(item.get("excludes") or [], f"{label}.excludes"),
|
||||||
|
"build_dir": f".site-publish/{name}/html",
|
||||||
|
"bucket": bucket,
|
||||||
|
"s3_endpoint": DEFAULT_S3_ENDPOINT,
|
||||||
|
"website_authority": authority,
|
||||||
|
"credentials": normalized_credentials,
|
||||||
|
"cache_rules": cache_rules,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _route(item, index):
|
||||||
|
label = f"routes[{index}]"
|
||||||
|
item = _mapping(item, label)
|
||||||
|
_known_keys(item, {"name", "path", "artifact", "access", "middlewares"}, label)
|
||||||
|
name = item.get("name")
|
||||||
|
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
|
||||||
|
raise ConfigError(f"{label}.name must be a DNS label")
|
||||||
|
access = _mapping(item.get("access"), f"{label}.access")
|
||||||
|
_known_keys(access, {"mode", "middleware"}, f"{label}.access")
|
||||||
|
mode, middleware = access.get("mode"), access.get("middleware")
|
||||||
|
if mode not in {"public", "protected"}:
|
||||||
|
raise ConfigError(f"{label}.access.mode must be public or protected")
|
||||||
|
if mode == "protected" and (not isinstance(middleware, str) or not MIDDLEWARE_RE.fullmatch(middleware)):
|
||||||
|
raise ConfigError(f"{label}.access.middleware is required for protected access")
|
||||||
|
if mode == "public" and middleware is not None:
|
||||||
|
raise ConfigError(f"{label}.access.middleware is forbidden for public access")
|
||||||
|
middlewares = _middlewares(item.get("middlewares") or [], f"{label}.middlewares")
|
||||||
|
if middleware in middlewares:
|
||||||
|
raise ConfigError(f"{label} repeats its access middleware")
|
||||||
|
artifact = item.get("artifact")
|
||||||
|
if not isinstance(artifact, str):
|
||||||
|
raise ConfigError(f"{label}.artifact must name an artifact")
|
||||||
|
return {
|
||||||
|
"name": name, "path": _route_path(item.get("path"), f"{label}.path"),
|
||||||
|
"artifact": artifact, "access": mode, "access_middleware": middleware,
|
||||||
"middlewares": middlewares,
|
"middlewares": middlewares,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_multi(cfg):
|
||||||
|
artifacts, routes = cfg["artifacts"], cfg["routes"]
|
||||||
|
artifact_names = [item["name"] for item in artifacts]
|
||||||
|
route_names = [item["name"] for item in routes]
|
||||||
|
route_paths = [item["path"] for item in routes]
|
||||||
|
if len(artifact_names) != len(set(artifact_names)):
|
||||||
|
raise ConfigError("artifact names must be unique")
|
||||||
|
if len(route_names) != len(set(route_names)):
|
||||||
|
raise ConfigError("route names must be unique")
|
||||||
|
if len(route_paths) != len(set(route_paths)):
|
||||||
|
raise ConfigError("route paths are ambiguous after normalization")
|
||||||
|
if "/" not in route_paths:
|
||||||
|
raise ConfigError("routes must declare a '/' catch-all")
|
||||||
|
artifact_by_name = {item["name"]: item for item in artifacts}
|
||||||
|
references = {name: [] for name in artifact_by_name}
|
||||||
|
for route in routes:
|
||||||
|
if route["artifact"] not in artifact_by_name:
|
||||||
|
raise ConfigError(f"route {route['name']} references unknown artifact {route['artifact']}")
|
||||||
|
references[route["artifact"]].append(route)
|
||||||
|
for name, used_by in references.items():
|
||||||
|
if len(used_by) != 1:
|
||||||
|
raise ConfigError(f"artifact {name} must be referenced by exactly one route")
|
||||||
|
buckets, authorities, credential_owners = {}, {}, {}
|
||||||
|
for route in routes:
|
||||||
|
artifact = artifact_by_name[route["artifact"]]
|
||||||
|
if artifact["bucket"] in buckets:
|
||||||
|
other_access, other_name = buckets[artifact["bucket"]]
|
||||||
|
if other_access != route["access"]:
|
||||||
|
raise ConfigError(f"bucket {artifact['bucket']} cannot be reused by protected and public routes")
|
||||||
|
raise ConfigError(f"bucket {artifact['bucket']} must belong to one artifact ({other_name})")
|
||||||
|
buckets[artifact["bucket"]] = (route["access"], artifact["name"])
|
||||||
|
if artifact["website_authority"] in authorities:
|
||||||
|
raise ConfigError(
|
||||||
|
f"website authority {artifact['website_authority']} must belong to one artifact"
|
||||||
|
)
|
||||||
|
authorities[artifact["website_authority"]] = artifact["name"]
|
||||||
|
for variable in artifact["credentials"].values():
|
||||||
|
if variable in credential_owners:
|
||||||
|
raise ConfigError(
|
||||||
|
f"publication credential {variable} is reused by artifacts "
|
||||||
|
f"{credential_owners[variable]} and {artifact['name']}"
|
||||||
|
)
|
||||||
|
credential_owners[variable] = artifact["name"]
|
||||||
|
for cache_rule in artifact["cache_rules"]:
|
||||||
|
directives = {part.strip().lower().split("=", 1)[0]
|
||||||
|
for part in cache_rule["cache_control"].split(",")}
|
||||||
|
if route["access"] == "protected" and "public" in directives:
|
||||||
|
raise ConfigError(f"protected route {route['name']} cannot use public cache policy")
|
||||||
|
if route["access"] == "protected" and not directives & {"private", "no-store"}:
|
||||||
|
raise ConfigError(f"protected route {route['name']} cache policy must be private or no-store")
|
||||||
|
if route["access"] == "protected" and "s-maxage" in directives:
|
||||||
|
raise ConfigError(f"protected route {route['name']} cannot use shared-cache max-age")
|
||||||
|
if route["access"] == "public" and "private" in directives:
|
||||||
|
raise ConfigError(f"public route {route['name']} cannot use private cache policy")
|
||||||
|
root = next(route for route in routes if route["path"] == "/")
|
||||||
|
if any(route["access"] == "protected" for route in routes) and root["access"] == "public":
|
||||||
|
raise ConfigError("a public '/' catch-all would expose unmatched protected content")
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_site_config(raw, site_name):
|
||||||
|
raw = _mapping(raw, "site.yaml")
|
||||||
|
domain = _hostname(raw.get("domain"), "domain")
|
||||||
|
if not isinstance(raw.get("enabled", True), bool):
|
||||||
|
raise ConfigError("enabled must be a boolean")
|
||||||
|
if ("artifacts" in raw) != ("routes" in raw):
|
||||||
|
raise ConfigError("artifacts and routes must be declared together")
|
||||||
|
if "artifacts" not in raw:
|
||||||
|
return _legacy_config(raw, site_name)
|
||||||
|
legacy_fields = {"type", "content_dir", "tidy", "excludes", "middlewares"} & set(raw)
|
||||||
|
if legacy_fields:
|
||||||
|
raise ConfigError("legacy fields cannot be mixed with artifacts/routes: " + ", ".join(sorted(legacy_fields)))
|
||||||
|
_known_keys(raw, {"domain", "aliases", "enabled", "artifacts", "routes"}, "site.yaml")
|
||||||
|
artifacts = [_artifact(item, index) for index, item in enumerate(_list(raw["artifacts"], "artifacts"))]
|
||||||
|
routes = [_route(item, index) for index, item in enumerate(_list(raw["routes"], "routes"))]
|
||||||
|
if not artifacts or not routes:
|
||||||
|
raise ConfigError("artifacts and routes must not be empty")
|
||||||
|
cfg = {
|
||||||
|
"version": 2, "compatibility": None, "domain": domain,
|
||||||
|
"aliases": _aliases(raw.get("aliases"), domain),
|
||||||
|
"enabled": raw.get("enabled", True),
|
||||||
|
"artifacts": sorted(artifacts, key=lambda item: item["name"]),
|
||||||
|
"routes": sorted(routes, key=lambda item: (-len(item["path"]), item["path"], item["name"])),
|
||||||
|
}
|
||||||
|
_validate_multi(cfg)
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
if len(f"{k8s_name(site_name)}-{route['name']}") > 63:
|
||||||
|
raise ConfigError(f"route {route['name']} makes the generated Service name exceed 63 characters")
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def validate_artifact_inputs(site_dir, cfg):
|
||||||
|
"""Reject source containment before a public or protected build starts."""
|
||||||
|
if cfg["compatibility"]:
|
||||||
|
return
|
||||||
|
root = Path(site_dir).resolve()
|
||||||
|
sources = []
|
||||||
|
for artifact in cfg["artifacts"]:
|
||||||
|
declared = root
|
||||||
|
for component in Path(artifact["content_dir"]).parts:
|
||||||
|
declared /= component
|
||||||
|
if declared.is_symlink():
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} content_dir contains symlink component: "
|
||||||
|
f"{declared.relative_to(root)}"
|
||||||
|
)
|
||||||
|
source = declared.resolve()
|
||||||
|
if source != root and root not in source.parents:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} content_dir resolves outside the repository"
|
||||||
|
)
|
||||||
|
if source.exists():
|
||||||
|
symlink = next((path for path in source.rglob("*") if path.is_symlink()), None)
|
||||||
|
if symlink is not None:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} build input contains symlink: "
|
||||||
|
f"{symlink.relative_to(root)}"
|
||||||
|
)
|
||||||
|
sources.append((artifact["name"], source))
|
||||||
|
for index, (name, source) in enumerate(sources):
|
||||||
|
for other_name, other_source in sources[index + 1:]:
|
||||||
|
if source == other_source or source in other_source.parents or other_source in source.parents:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact build inputs overlap after resolution: {name} and {other_name}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_site_yaml(site_dir, site_name=None):
|
||||||
|
path = Path(site_dir) / "site.yaml"
|
||||||
|
if not path.exists():
|
||||||
|
die("site.yaml not found in repo root")
|
||||||
|
if site_name is None:
|
||||||
|
repo = os.environ.get("SITE_REPO", "")
|
||||||
|
site_name = repo.split("/", 1)[-1] if "/" in repo else Path(site_dir).name
|
||||||
|
try:
|
||||||
|
with open(path) as stream:
|
||||||
|
cfg = normalize_site_config(yaml.safe_load(stream), site_name)
|
||||||
|
except (ConfigError, yaml.YAMLError) as exc:
|
||||||
|
die(str(exc))
|
||||||
print("Site config:")
|
print("Site config:")
|
||||||
for k, v in site.items():
|
print(f" domain: {cfg['domain']}")
|
||||||
print(f" {k}: {v}")
|
print(f" contract: {cfg['compatibility'] or 'split-surface-v2'}")
|
||||||
return site
|
print(f" artifacts: {[item['name'] for item in cfg['artifacts']]}")
|
||||||
|
print(f" routes: {[(item['path'], item['access']) for item in cfg['routes']]}")
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
def clone_apps(token):
|
def clone_apps(token):
|
||||||
|
"""Clone Apps without placing the credential in argv or output."""
|
||||||
user = env("CI_BOT_USER", "ci-bot")
|
user = env("CI_BOT_USER", "ci-bot")
|
||||||
apps_dir = Path("/tmp/apps-deploy")
|
apps_dir = Path("/tmp/apps-deploy")
|
||||||
if apps_dir.exists():
|
if apps_dir.exists():
|
||||||
shutil.rmtree(apps_dir)
|
shutil.rmtree(apps_dir)
|
||||||
run(f"git clone --depth 1 https://{user}:{token}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}")
|
clone_env = git_auth_env(token)
|
||||||
run(f"git -C {apps_dir} config user.name {user}")
|
url = f"https://{user}@{GITEA_HOST}/{APPS_REPO}.git"
|
||||||
run(f"git -C {apps_dir} config user.email {user}@fritzlab.net")
|
run(["git", "clone", "--depth", "1", url, str(apps_dir)],
|
||||||
|
display=f"git clone --depth 1 https://{user}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}", env=clone_env)
|
||||||
|
run(["git", "-C", str(apps_dir), "config", "user.name", user])
|
||||||
|
run(["git", "-C", str(apps_dir), "config", "user.email", f"{user}@fritzlab.net"])
|
||||||
return apps_dir
|
return apps_dir
|
||||||
|
|
||||||
|
|
||||||
def render_templates(action_dir, template_vars, app_dir, manifests_dir):
|
def render_templates(action_dir, template_vars, app_dir, manifests_dir):
|
||||||
"""Render Jinja2 templates for a static-content site."""
|
"""Render one certificate and deterministic per-route resources."""
|
||||||
templates_dir = Path(action_dir) / "templates"
|
jinja_env = Environment(loader=FileSystemLoader(str(Path(action_dir) / "templates")),
|
||||||
jinja_env = Environment(
|
keep_trailing_newline=True, undefined=StrictUndefined)
|
||||||
loader=FileSystemLoader(str(templates_dir)),
|
manifests_dir.mkdir(parents=True, exist_ok=True)
|
||||||
keep_trailing_newline=True,
|
for child in manifests_dir.iterdir():
|
||||||
)
|
if child.is_file() and child.suffix in {".yaml", ".yml"}:
|
||||||
|
child.unlink()
|
||||||
tmpl_names = ["app.yaml.j2", "certificate.yaml.j2", "ingress.yaml.j2",
|
route_files = []
|
||||||
"kustomization.yaml.j2", "service.yaml.j2"]
|
for route in template_vars["routes"]:
|
||||||
|
stem = "" if template_vars["compatibility"] else f"-{route['name']}"
|
||||||
for tmpl_name in tmpl_names:
|
for kind in ("service", "ingress"):
|
||||||
tmpl = jinja_env.get_template(tmpl_name)
|
out_name = f"{kind}{stem}.yaml"
|
||||||
rendered = tmpl.render(**template_vars)
|
destination = manifests_dir / out_name
|
||||||
out_name = tmpl_name.replace(".j2", "")
|
destination.write_text(jinja_env.get_template(f"{kind}.yaml.j2").render(
|
||||||
dest = app_dir / out_name if tmpl_name == "app.yaml.j2" else manifests_dir / out_name
|
**template_vars, route=route
|
||||||
dest.write_text(rendered)
|
))
|
||||||
print(f" Rendered {tmpl_name} -> {dest}")
|
route_files.append(out_name)
|
||||||
|
print(f" Rendered {kind}.yaml.j2 -> {destination}")
|
||||||
|
common_vars = {**template_vars, "route_files": route_files}
|
||||||
|
for out_name, destination in {
|
||||||
|
"certificate.yaml": manifests_dir / "certificate.yaml",
|
||||||
|
"kustomization.yaml": manifests_dir / "kustomization.yaml",
|
||||||
|
"app.yaml": app_dir / "app.yaml",
|
||||||
|
}.items():
|
||||||
|
destination.write_text(jinja_env.get_template(f"{out_name}.j2").render(**common_vars))
|
||||||
|
print(f" Rendered {out_name}.j2 -> {destination}")
|
||||||
|
|
||||||
|
|
||||||
def commit_and_push(apps_dir, message):
|
def git_auth_env(token):
|
||||||
run(f"git -C {apps_dir} add -A")
|
"""Return credential-safe Git authentication shared by clone and push."""
|
||||||
result = subprocess.run(
|
auth_env = os.environ.copy()
|
||||||
f"git -C {apps_dir} diff --cached --quiet",
|
auth_env["CI_BOT_TOKEN"] = token
|
||||||
shell=True, check=False,
|
auth_env["GIT_ASKPASS"] = str(Path(__file__).with_name("git-askpass.sh"))
|
||||||
)
|
auth_env["GIT_TERMINAL_PROMPT"] = "0"
|
||||||
|
return auth_env
|
||||||
|
|
||||||
|
|
||||||
|
def commit_and_push(apps_dir, message, token=None):
|
||||||
|
run(["git", "-C", str(apps_dir), "add", "-A"])
|
||||||
|
result = subprocess.run(["git", "-C", str(apps_dir), "diff", "--cached", "--quiet"], check=False)
|
||||||
if result.returncode == 0:
|
if result.returncode == 0:
|
||||||
print("No manifest changes to commit")
|
print("No manifest changes to commit")
|
||||||
return False
|
return False
|
||||||
run(f"git -C {apps_dir} commit -m '{message}'")
|
run(["git", "-C", str(apps_dir), "commit", "-m", message])
|
||||||
run(f"git -C {apps_dir} push")
|
push_env = git_auth_env(token) if token else None
|
||||||
|
run(["git", "-C", str(apps_dir), "push"], env=push_env)
|
||||||
print("Manifests pushed — ArgoCD will sync")
|
print("Manifests pushed — ArgoCD will sync")
|
||||||
return True
|
return True
|
||||||
|
|||||||
@@ -1,12 +1,10 @@
|
|||||||
apiVersion: networking.k8s.io/v1
|
apiVersion: networking.k8s.io/v1
|
||||||
kind: Ingress
|
kind: Ingress
|
||||||
metadata:
|
metadata:
|
||||||
name: {{ site_k8s }}
|
name: {{ route.resource_name }}
|
||||||
namespace: {{ namespace }}
|
namespace: {{ namespace }}
|
||||||
{%- if site_type != "docker" %}
|
|
||||||
annotations:
|
annotations:
|
||||||
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% for m in middlewares %},{{ m }}@file{% endfor %}
|
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% if route.access_middleware %},{{ route.access_middleware }}@file{% endif %}{% for m in route.middlewares %},{{ m }}@file{% endfor %}
|
||||||
{%- endif %}
|
|
||||||
spec:
|
spec:
|
||||||
ingressClassName: traefik
|
ingressClassName: traefik
|
||||||
tls:
|
tls:
|
||||||
@@ -20,22 +18,22 @@ spec:
|
|||||||
- host: {{ domain }}
|
- host: {{ domain }}
|
||||||
http:
|
http:
|
||||||
paths:
|
paths:
|
||||||
- path: /
|
- path: {{ route.path }}
|
||||||
pathType: Prefix
|
pathType: Prefix
|
||||||
backend:
|
backend:
|
||||||
service:
|
service:
|
||||||
name: {{ site_k8s }}
|
name: {{ route.resource_name }}
|
||||||
port:
|
port:
|
||||||
number: 80
|
number: 80
|
||||||
{%- for alias in aliases %}
|
{%- for alias in aliases %}
|
||||||
- host: {{ alias }}
|
- host: {{ alias }}
|
||||||
http:
|
http:
|
||||||
paths:
|
paths:
|
||||||
- path: /
|
- path: {{ route.path }}
|
||||||
pathType: Prefix
|
pathType: Prefix
|
||||||
backend:
|
backend:
|
||||||
service:
|
service:
|
||||||
name: {{ site_k8s }}
|
name: {{ route.resource_name }}
|
||||||
port:
|
port:
|
||||||
number: 80
|
number: 80
|
||||||
{%- endfor %}
|
{%- endfor %}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
kind: Kustomization
|
kind: Kustomization
|
||||||
resources:
|
resources:
|
||||||
- service.yaml
|
{% for route_file in route_files -%}
|
||||||
- ingress.yaml
|
- {{ route_file }}
|
||||||
|
{% endfor -%}
|
||||||
- certificate.yaml
|
- certificate.yaml
|
||||||
|
|||||||
@@ -1,11 +1,15 @@
|
|||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Service
|
kind: Service
|
||||||
metadata:
|
metadata:
|
||||||
name: {{ site_k8s }}
|
name: {{ route.resource_name }}
|
||||||
namespace: {{ namespace }}
|
namespace: {{ namespace }}
|
||||||
|
{%- if not compatibility %}
|
||||||
|
annotations:
|
||||||
|
traefik.ingress.kubernetes.io/service.passhostheader: "false"
|
||||||
|
{%- endif %}
|
||||||
spec:
|
spec:
|
||||||
type: ExternalName
|
type: ExternalName
|
||||||
externalName: garage.storage.svc.k8s.sjc001.fritzlab.net
|
externalName: {{ route.artifact_config.website_authority }}
|
||||||
ports:
|
ports:
|
||||||
- port: 80
|
- port: 80
|
||||||
targetPort: 80
|
targetPort: 80
|
||||||
|
|||||||
Vendored
+9
@@ -0,0 +1,9 @@
|
|||||||
|
domain: example.fritzlab.net
|
||||||
|
type: static
|
||||||
|
content_dir: html
|
||||||
|
aliases:
|
||||||
|
- www.example.fritzlab.net
|
||||||
|
middlewares:
|
||||||
|
- response-headers
|
||||||
|
excludes:
|
||||||
|
- media/*
|
||||||
Vendored
+42
@@ -0,0 +1,42 @@
|
|||||||
|
domain: baseline.fritzlab.net
|
||||||
|
artifacts:
|
||||||
|
- name: portal
|
||||||
|
type: static
|
||||||
|
content_dir: portal/build
|
||||||
|
publish:
|
||||||
|
bucket: baseline-portal
|
||||||
|
credentials:
|
||||||
|
access_key_env: PORTAL_S3_ACCESS_KEY
|
||||||
|
secret_key_env: PORTAL_S3_SECRET_KEY
|
||||||
|
cache:
|
||||||
|
rules:
|
||||||
|
- path: /
|
||||||
|
cache_control: private, no-store
|
||||||
|
- name: distributions
|
||||||
|
type: static
|
||||||
|
content_dir: dist
|
||||||
|
publish:
|
||||||
|
bucket: baseline-dist
|
||||||
|
credentials:
|
||||||
|
access_key_env: DIST_S3_ACCESS_KEY
|
||||||
|
secret_key_env: DIST_S3_SECRET_KEY
|
||||||
|
cache:
|
||||||
|
rules:
|
||||||
|
- path: /
|
||||||
|
cache_control: public, max-age=0, must-revalidate
|
||||||
|
- path: releases
|
||||||
|
cache_control: public, max-age=31536000, immutable
|
||||||
|
- path: channels
|
||||||
|
cache_control: public, max-age=0, must-revalidate
|
||||||
|
routes:
|
||||||
|
- name: portal
|
||||||
|
path: /
|
||||||
|
artifact: portal
|
||||||
|
access:
|
||||||
|
mode: protected
|
||||||
|
middleware: authentik-forwardauth
|
||||||
|
- name: distributions
|
||||||
|
path: /dist
|
||||||
|
artifact: distributions
|
||||||
|
access:
|
||||||
|
mode: public
|
||||||
@@ -0,0 +1,698 @@
|
|||||||
|
import base64
|
||||||
|
import copy
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import unittest
|
||||||
|
from contextlib import redirect_stderr, redirect_stdout
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT / "scripts"))
|
||||||
|
|
||||||
|
import deploy
|
||||||
|
import build
|
||||||
|
import utils
|
||||||
|
from utils import ConfigError, normalize_site_config, validate_artifact_inputs
|
||||||
|
|
||||||
|
|
||||||
|
def fixture(name):
|
||||||
|
return yaml.safe_load((ROOT / "tests" / "fixtures" / name).read_text())
|
||||||
|
|
||||||
|
|
||||||
|
class IPv6GitHTTPServer(ThreadingHTTPServer):
|
||||||
|
address_family = socket.AF_INET6
|
||||||
|
|
||||||
|
|
||||||
|
class AuthenticatedGitHandler(BaseHTTPRequestHandler):
|
||||||
|
project_root = None
|
||||||
|
expected_authorization = None
|
||||||
|
|
||||||
|
def log_message(self, _format, *_args):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
self._git_backend()
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
self._git_backend()
|
||||||
|
|
||||||
|
def _git_backend(self):
|
||||||
|
if self.headers.get("Authorization") != self.expected_authorization:
|
||||||
|
self.send_response(401)
|
||||||
|
self.send_header("WWW-Authenticate", 'Basic realm="test"')
|
||||||
|
self.end_headers()
|
||||||
|
return
|
||||||
|
parsed = urlsplit(self.path)
|
||||||
|
length = int(self.headers.get("Content-Length", "0"))
|
||||||
|
request_body = self.rfile.read(length) if length else b""
|
||||||
|
backend_env = os.environ.copy()
|
||||||
|
backend_env.update({
|
||||||
|
"GIT_PROJECT_ROOT": str(self.project_root),
|
||||||
|
"GIT_HTTP_EXPORT_ALL": "1",
|
||||||
|
"PATH_INFO": parsed.path,
|
||||||
|
"QUERY_STRING": parsed.query,
|
||||||
|
"REQUEST_METHOD": self.command,
|
||||||
|
"CONTENT_TYPE": self.headers.get("Content-Type", ""),
|
||||||
|
"CONTENT_LENGTH": str(length),
|
||||||
|
"REMOTE_USER": "ci-bot",
|
||||||
|
"REMOTE_ADDR": "::1",
|
||||||
|
"GATEWAY_INTERFACE": "CGI/1.1",
|
||||||
|
"SERVER_PROTOCOL": "HTTP/1.1",
|
||||||
|
})
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "http-backend"], input=request_body, env=backend_env,
|
||||||
|
capture_output=True, check=True,
|
||||||
|
)
|
||||||
|
raw_headers, response_body = result.stdout.split(b"\r\n\r\n", 1)
|
||||||
|
headers, status = [], 200
|
||||||
|
for line in raw_headers.decode().split("\r\n"):
|
||||||
|
name, value = line.split(":", 1)
|
||||||
|
if name.lower() == "status":
|
||||||
|
status = int(value.strip().split(" ", 1)[0])
|
||||||
|
else:
|
||||||
|
headers.append((name, value.strip()))
|
||||||
|
self.send_response(status)
|
||||||
|
for name, value in headers:
|
||||||
|
self.send_header(name, value)
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(response_body)
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigContractTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.raw = fixture("split-site.yaml")
|
||||||
|
|
||||||
|
def assert_invalid(self, mutate, message):
|
||||||
|
raw = copy.deepcopy(self.raw)
|
||||||
|
mutate(raw)
|
||||||
|
with self.assertRaisesRegex(ConfigError, message):
|
||||||
|
normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
|
||||||
|
def test_legacy_normalizes_to_explicit_compatibility_surface(self):
|
||||||
|
cfg = normalize_site_config(fixture("legacy-site.yaml"), "example.fritzlab.net")
|
||||||
|
self.assertEqual("single-surface-v1", cfg["compatibility"])
|
||||||
|
self.assertEqual("build/html", cfg["artifacts"][0]["build_dir"])
|
||||||
|
self.assertEqual("example.fritzlab.net", cfg["artifacts"][0]["bucket"])
|
||||||
|
self.assertEqual("/", cfg["routes"][0]["path"])
|
||||||
|
self.assertEqual("legacy", cfg["routes"][0]["access"])
|
||||||
|
|
||||||
|
def test_routes_are_sorted_longest_prefix_first(self):
|
||||||
|
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
|
||||||
|
self.assertEqual(["/dist", "/"], [route["path"] for route in cfg["routes"]])
|
||||||
|
credentials = {artifact["name"]: artifact["credentials"] for artifact in cfg["artifacts"]}
|
||||||
|
self.assertNotEqual(credentials["distributions"], credentials["portal"])
|
||||||
|
|
||||||
|
def test_equivalent_route_paths_are_ambiguous(self):
|
||||||
|
self.assert_invalid(lambda raw: raw["routes"].append({
|
||||||
|
"name": "duplicate", "path": "/dist/", "artifact": "portal",
|
||||||
|
"access": {"mode": "protected", "middleware": "authentik-forwardauth"},
|
||||||
|
}), "ambiguous")
|
||||||
|
|
||||||
|
def test_catch_all_is_required(self):
|
||||||
|
self.assert_invalid(lambda raw: raw["routes"].__setitem__(0, {
|
||||||
|
**raw["routes"][0], "path": "/portal"
|
||||||
|
}), "catch-all")
|
||||||
|
|
||||||
|
def test_public_catch_all_is_rejected_when_any_route_is_protected(self):
|
||||||
|
def mutate(raw):
|
||||||
|
raw["routes"][0]["access"] = {"mode": "public"}
|
||||||
|
raw["routes"][1]["access"] = {
|
||||||
|
"mode": "protected", "middleware": "authentik-forwardauth"
|
||||||
|
}
|
||||||
|
raw["artifacts"][0]["cache"]["rules"][0]["cache_control"] = (
|
||||||
|
"public, max-age=0, must-revalidate"
|
||||||
|
)
|
||||||
|
for rule in raw["artifacts"][1]["cache"]["rules"]:
|
||||||
|
rule["cache_control"] = "private, no-store"
|
||||||
|
self.assert_invalid(mutate, "public '/' catch-all")
|
||||||
|
|
||||||
|
def test_protected_route_requires_middleware(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["routes"][0].__setitem__("access", {"mode": "protected"}),
|
||||||
|
"middleware is required",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_bucket_cannot_cross_access_boundary(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["artifacts"][0]["publish"].__setitem__("bucket", "baseline-dist"),
|
||||||
|
"protected and public",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_publication_credentials_cannot_be_reused(self):
|
||||||
|
def mutate(raw):
|
||||||
|
raw["artifacts"][0]["publish"]["credentials"] = copy.deepcopy(
|
||||||
|
raw["artifacts"][1]["publish"]["credentials"]
|
||||||
|
)
|
||||||
|
self.assert_invalid(mutate, "publication credential DIST_S3_ACCESS_KEY is reused")
|
||||||
|
|
||||||
|
def test_cache_directive_contradiction_is_rejected(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["artifacts"][1]["cache"]["rules"][1].__setitem__(
|
||||||
|
"cache_control", "public, max-age=31536000, immutable, must-revalidate"
|
||||||
|
),
|
||||||
|
"immutable requires",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_cache_policy_cannot_nest_below_immutable_path(self):
|
||||||
|
def mutate(raw):
|
||||||
|
raw["artifacts"][1]["cache"]["rules"].append({
|
||||||
|
"path": "releases/candidates",
|
||||||
|
"cache_control": "public, max-age=0, must-revalidate",
|
||||||
|
})
|
||||||
|
self.assert_invalid(mutate, "cannot nest another policy under immutable /releases")
|
||||||
|
|
||||||
|
def test_public_cache_is_rejected_on_protected_route(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["artifacts"][0]["cache"]["rules"][0].__setitem__(
|
||||||
|
"cache_control", "public, max-age=0, must-revalidate"
|
||||||
|
),
|
||||||
|
"protected route portal",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_legacy_and_split_fields_cannot_mix(self):
|
||||||
|
self.assert_invalid(lambda raw: raw.__setitem__("type", "static"), "cannot be mixed")
|
||||||
|
|
||||||
|
def test_unknown_split_field_is_rejected(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["artifacts"][0].__setitem__("storage_bucket", "typo"),
|
||||||
|
"unknown fields: storage_bucket",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_backend_authority_and_endpoint_are_derived(self):
|
||||||
|
for field, value in (
|
||||||
|
("endpoint", "https://attacker.example"),
|
||||||
|
("website_authority", "internal-api.default.svc.k8s.sjc001.fritzlab.net"),
|
||||||
|
):
|
||||||
|
with self.subTest(field=field):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw, field=field, value=value: raw["artifacts"][0]["publish"].__setitem__(field, value),
|
||||||
|
f"unknown fields: {field}",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_publication_credentials_use_dedicated_matched_names(self):
|
||||||
|
self.assert_invalid(
|
||||||
|
lambda raw: raw["artifacts"][0]["publish"]["credentials"].__setitem__(
|
||||||
|
"access_key_env", "CI_BOT_TOKEN"
|
||||||
|
),
|
||||||
|
"must be a matched <NAME>_S3_ACCESS_KEY",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_resolved_build_inputs_must_be_pairwise_disjoint(self):
|
||||||
|
raw = copy.deepcopy(self.raw)
|
||||||
|
raw["artifacts"][1]["content_dir"] = ""
|
||||||
|
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
(root / "portal" / "build").mkdir(parents=True)
|
||||||
|
with self.assertRaisesRegex(ConfigError, "build inputs overlap after resolution"):
|
||||||
|
validate_artifact_inputs(root, cfg)
|
||||||
|
|
||||||
|
def test_descendant_symlink_cannot_cross_artifact_boundary(self):
|
||||||
|
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
(root / "dist").mkdir()
|
||||||
|
(root / "portal" / "build").mkdir(parents=True)
|
||||||
|
(root / "portal" / "build" / "private.txt").write_text("private")
|
||||||
|
(root / "dist" / "portal-link").symlink_to(root / "portal" / "build")
|
||||||
|
with self.assertRaisesRegex(ConfigError, "build input contains symlink"):
|
||||||
|
validate_artifact_inputs(root, cfg)
|
||||||
|
|
||||||
|
def test_artifact_root_symlink_is_rejected_before_resolution(self):
|
||||||
|
cfg = normalize_site_config(self.raw, "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
(root / "dist-real").mkdir()
|
||||||
|
(root / "dist").symlink_to(root / "dist-real")
|
||||||
|
(root / "portal" / "build").mkdir(parents=True)
|
||||||
|
with self.assertRaisesRegex(ConfigError, "content_dir contains symlink component"):
|
||||||
|
validate_artifact_inputs(root, cfg)
|
||||||
|
|
||||||
|
|
||||||
|
class GenerationTests(unittest.TestCase):
|
||||||
|
def render(self, raw):
|
||||||
|
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
tmp = tempfile.TemporaryDirectory()
|
||||||
|
root = Path(tmp.name)
|
||||||
|
app_dir = root / "app"
|
||||||
|
manifests = app_dir / "manifests"
|
||||||
|
app_dir.mkdir(parents=True)
|
||||||
|
deploy.render_site_manifests(
|
||||||
|
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg
|
||||||
|
)
|
||||||
|
files = {path.relative_to(app_dir).as_posix(): path.read_text()
|
||||||
|
for path in sorted(app_dir.rglob("*.yaml"))}
|
||||||
|
return tmp, app_dir, files
|
||||||
|
|
||||||
|
def test_split_fixture_generates_per_route_resources_and_one_certificate(self):
|
||||||
|
tmp, _, files = self.render(fixture("split-site.yaml"))
|
||||||
|
self.addCleanup(tmp.cleanup)
|
||||||
|
self.assertEqual({
|
||||||
|
"app.yaml", "manifests/certificate.yaml", "manifests/ingress-distributions.yaml",
|
||||||
|
"manifests/ingress-portal.yaml", "manifests/kustomization.yaml",
|
||||||
|
"manifests/service-distributions.yaml", "manifests/service-portal.yaml",
|
||||||
|
}, set(files))
|
||||||
|
for content in files.values():
|
||||||
|
self.assertIsNotNone(yaml.safe_load(content))
|
||||||
|
self.assertIn("path: /dist", files["manifests/ingress-distributions.yaml"])
|
||||||
|
self.assertNotIn("authentik-forwardauth", files["manifests/ingress-distributions.yaml"])
|
||||||
|
self.assertIn("authentik-forwardauth@file", files["manifests/ingress-portal.yaml"])
|
||||||
|
self.assertIn('service.passhostheader: "false"', files["manifests/service-portal.yaml"])
|
||||||
|
self.assertNotIn("passhostheader", files["manifests/ingress-portal.yaml"])
|
||||||
|
self.assertIn("baseline-dist.web.sjc001.fritzlab.net", files["manifests/service-distributions.yaml"])
|
||||||
|
|
||||||
|
def test_history_keeps_yaml_ambiguous_artifact_names_as_strings(self):
|
||||||
|
raw = fixture("split-site.yaml")
|
||||||
|
raw["artifacts"][0]["name"] = "yes"
|
||||||
|
raw["routes"][0]["artifact"] = "yes"
|
||||||
|
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
contracts = deploy.next_route_contracts(cfg, {})
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
app_dir = Path(tmp)
|
||||||
|
deploy.write_route_contracts(app_dir, contracts)
|
||||||
|
restored = deploy.previous_route_contracts(app_dir)
|
||||||
|
self.assertEqual("yes", restored["baseline-portal"]["artifact"])
|
||||||
|
|
||||||
|
def test_generation_is_deterministic_when_input_lists_are_reversed(self):
|
||||||
|
raw = fixture("split-site.yaml")
|
||||||
|
first_tmp, _, first = self.render(raw)
|
||||||
|
self.addCleanup(first_tmp.cleanup)
|
||||||
|
raw["artifacts"].reverse()
|
||||||
|
raw["routes"].reverse()
|
||||||
|
second_tmp, _, second = self.render(raw)
|
||||||
|
self.addCleanup(second_tmp.cleanup)
|
||||||
|
self.assertEqual(first, second)
|
||||||
|
|
||||||
|
def test_stale_route_manifests_are_removed(self):
|
||||||
|
raw = fixture("split-site.yaml")
|
||||||
|
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
app_dir = Path(tmp) / "app"
|
||||||
|
manifests = app_dir / "manifests"
|
||||||
|
manifests.mkdir(parents=True)
|
||||||
|
stale = manifests / "ingress-removed.yaml"
|
||||||
|
stale.write_text("stale\n")
|
||||||
|
deploy.render_site_manifests("baseline.fritzlab.net", ROOT, app_dir, manifests, cfg)
|
||||||
|
self.assertFalse(stale.exists())
|
||||||
|
|
||||||
|
def test_legacy_names_and_garage_s3_target_are_preserved(self):
|
||||||
|
tmp, app_dir, files = self.render(fixture("legacy-site.yaml"))
|
||||||
|
self.addCleanup(tmp.cleanup)
|
||||||
|
self.assertIn("manifests/service.yaml", files)
|
||||||
|
self.assertIn("manifests/ingress.yaml", files)
|
||||||
|
self.assertIn("garage-s3.storage.svc.k8s.sjc001.fritzlab.net", files["manifests/service.yaml"])
|
||||||
|
self.assertNotIn("passhostheader", files["manifests/ingress.yaml"])
|
||||||
|
self.assertNotIn("site-publish.fritzlab.net", files["manifests/ingress.yaml"])
|
||||||
|
self.assertEqual({}, deploy.previous_route_contracts(app_dir))
|
||||||
|
|
||||||
|
|
||||||
|
class BuildTests(unittest.TestCase):
|
||||||
|
def test_artifacts_build_independently_without_clobbering_siblings(self):
|
||||||
|
raw = fixture("split-site.yaml")
|
||||||
|
for artifact in raw["artifacts"]:
|
||||||
|
artifact["tidy"] = False
|
||||||
|
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
(root / "dist").mkdir()
|
||||||
|
(root / "dist" / "bundle.js").write_text("bundle")
|
||||||
|
(root / "portal" / "build").mkdir(parents=True)
|
||||||
|
(root / "portal" / "build" / "index.html").write_text("portal")
|
||||||
|
for artifact in cfg["artifacts"]:
|
||||||
|
build.build_artifact(root, artifact)
|
||||||
|
self.assertEqual(
|
||||||
|
"bundle", (root / ".site-publish/distributions/html/bundle.js").read_text()
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
"portal", (root / ".site-publish/portal/html/index.html").read_text()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class PublishingTests(unittest.TestCase):
|
||||||
|
def test_apps_clone_never_places_token_in_argv_or_log(self):
|
||||||
|
calls = []
|
||||||
|
secret = "clone-secret-must-not-appear"
|
||||||
|
with patch.dict(os.environ, {"CI_BOT_USER": "ci-bot"}, clear=False), \
|
||||||
|
patch.object(utils, "run", side_effect=lambda command, **kwargs: calls.append((command, kwargs))), \
|
||||||
|
patch.object(utils.shutil, "rmtree"), redirect_stdout(io.StringIO()) as output:
|
||||||
|
utils.clone_apps(secret)
|
||||||
|
self.assertNotIn(secret, output.getvalue())
|
||||||
|
for command, kwargs in calls:
|
||||||
|
self.assertNotIn(secret, " ".join(command))
|
||||||
|
self.assertNotIn(secret, kwargs.get("display", ""))
|
||||||
|
|
||||||
|
def test_askpass_authenticates_clone_and_push_round_trip(self):
|
||||||
|
token = "round-trip-token"
|
||||||
|
expected = "Basic " + base64.b64encode(f"ci-bot:{token}".encode()).decode()
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
bare = root / "repo.git"
|
||||||
|
seed = root / "seed"
|
||||||
|
checkout = root / "checkout"
|
||||||
|
subprocess.run(["git", "init", "--bare", "--initial-branch=main", str(bare)], check=True,
|
||||||
|
stdout=subprocess.DEVNULL)
|
||||||
|
subprocess.run(["git", "-C", str(bare), "config", "http.receivepack", "true"], check=True)
|
||||||
|
subprocess.run(["git", "init", "--initial-branch=main", str(seed)], check=True,
|
||||||
|
stdout=subprocess.DEVNULL)
|
||||||
|
subprocess.run(["git", "-C", str(seed), "config", "user.name", "Test"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(seed), "config", "user.email", "test@example.invalid"], check=True)
|
||||||
|
(seed / "README.md").write_text("seed\n")
|
||||||
|
subprocess.run(["git", "-C", str(seed), "add", "README.md"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(seed), "commit", "-m", "seed"], check=True,
|
||||||
|
stdout=subprocess.DEVNULL)
|
||||||
|
subprocess.run(["git", "-C", str(seed), "push", str(bare), "main"], check=True,
|
||||||
|
stdout=subprocess.DEVNULL)
|
||||||
|
|
||||||
|
handler = type("GitHandler", (AuthenticatedGitHandler,), {
|
||||||
|
"project_root": root, "expected_authorization": expected,
|
||||||
|
})
|
||||||
|
server = IPv6GitHTTPServer(("::1", 0), handler)
|
||||||
|
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||||
|
thread.start()
|
||||||
|
try:
|
||||||
|
url = f"http://ci-bot@[::1]:{server.server_port}/repo.git"
|
||||||
|
with patch.dict(os.environ, {"NO_PROXY": "::1,[::1]", "no_proxy": "::1,[::1]"}, clear=False):
|
||||||
|
subprocess.run(
|
||||||
|
["git", "clone", url, str(checkout)], env=utils.git_auth_env(token),
|
||||||
|
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||||
|
)
|
||||||
|
subprocess.run(["git", "-C", str(checkout), "config", "user.name", "Test"], check=True)
|
||||||
|
subprocess.run(["git", "-C", str(checkout), "config", "user.email", "test@example.invalid"],
|
||||||
|
check=True)
|
||||||
|
(checkout / "roundtrip.txt").write_text("authenticated\n")
|
||||||
|
utils.commit_and_push(checkout, "authenticated round trip", token)
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
server.server_close()
|
||||||
|
thread.join(timeout=5)
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "--git-dir", str(bare), "show", "main:roundtrip.txt"],
|
||||||
|
check=True, text=True, capture_output=True,
|
||||||
|
)
|
||||||
|
self.assertEqual("authenticated\n", result.stdout)
|
||||||
|
|
||||||
|
def test_cache_headers_credentials_and_route_prefix_are_separate(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
|
||||||
|
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
html = root / artifact["build_dir"]
|
||||||
|
(html / "releases").mkdir(parents=True)
|
||||||
|
(html / "channels").mkdir()
|
||||||
|
(html / "releases" / "1.0.js").write_text("release")
|
||||||
|
(html / "channels" / "stable.json").write_text("channel")
|
||||||
|
commands, events = [], []
|
||||||
|
|
||||||
|
def capture(command, **kwargs):
|
||||||
|
commands.append((command, kwargs["env"]))
|
||||||
|
events.append("mutable")
|
||||||
|
|
||||||
|
def publish_immutable(*_args):
|
||||||
|
events.append("immutable")
|
||||||
|
|
||||||
|
secret = "secret-must-not-appear"
|
||||||
|
output = io.StringIO()
|
||||||
|
with patch.dict(os.environ, {
|
||||||
|
"DIST_S3_ACCESS_KEY": "dist-key", "DIST_S3_SECRET_KEY": secret
|
||||||
|
}, clear=False), patch.object(deploy, "run", side_effect=capture), \
|
||||||
|
patch.object(deploy, "publish_immutable_rule", side_effect=publish_immutable) as immutable_publish, \
|
||||||
|
redirect_stdout(output):
|
||||||
|
deploy.publish_route_immutables(artifact, route, root)
|
||||||
|
deploy.s3_sync(artifact, route, root)
|
||||||
|
|
||||||
|
self.assertTrue(all(secret not in " ".join(command) for command, _ in commands))
|
||||||
|
self.assertEqual("immutable", events[0])
|
||||||
|
self.assertNotIn(secret, output.getvalue())
|
||||||
|
self.assertTrue(all(call_env["AWS_ACCESS_KEY_ID"] == "dist-key" for _, call_env in commands))
|
||||||
|
self.assertTrue(all("DIST_S3_SECRET_KEY" not in call_env for _, call_env in commands))
|
||||||
|
rendered = [" ".join(command) for command, _ in commands]
|
||||||
|
self.assertIn("s3://baseline-dist/dist/", rendered[0])
|
||||||
|
self.assertIn("releases/*", rendered[0])
|
||||||
|
self.assertNotIn("--delete", rendered[0])
|
||||||
|
self.assertIn("--delete", rendered[1])
|
||||||
|
self.assertTrue(all("s3://baseline-dist/dist/" in command for command in rendered[1:]))
|
||||||
|
self.assertTrue(any("channels/" in command and
|
||||||
|
"public, max-age=0, must-revalidate" in command
|
||||||
|
for command in rendered))
|
||||||
|
immutable_publish.assert_called_once()
|
||||||
|
self.assertEqual(
|
||||||
|
"public, max-age=31536000, immutable",
|
||||||
|
immutable_publish.call_args.args[2]["cache_control"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_root_move_preserves_only_actual_retired_immutable_prefix(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
|
||||||
|
route = {**next(item for item in cfg["routes"] if item["artifact"] == "distributions"),
|
||||||
|
"path": "/"}
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
html = root / artifact["build_dir"]
|
||||||
|
(html / "releases").mkdir(parents=True)
|
||||||
|
(html / "channels").mkdir()
|
||||||
|
(html / "docs" / "releases").mkdir(parents=True)
|
||||||
|
(html / "channels" / "stable.json").write_text("channel")
|
||||||
|
(html / "docs" / "releases" / "index.html").write_text("mutable")
|
||||||
|
commands = []
|
||||||
|
with patch.dict(os.environ, {
|
||||||
|
"DIST_S3_ACCESS_KEY": "dist-key", "DIST_S3_SECRET_KEY": "dist-secret"
|
||||||
|
}, clear=False), patch.object(
|
||||||
|
deploy, "run", side_effect=lambda command, **_: commands.append(command)
|
||||||
|
):
|
||||||
|
deploy.s3_sync(artifact, route, root, previous_contract={
|
||||||
|
"path": "/foo", "access": "public", "artifact": "distributions",
|
||||||
|
"immutable_prefixes": ["foo/releases"],
|
||||||
|
})
|
||||||
|
rendered = [" ".join(command) for command in commands]
|
||||||
|
self.assertTrue(all("foo/releases/*" in command for command in rendered[:2]))
|
||||||
|
self.assertTrue(all("*/releases/*" not in command for command in rendered))
|
||||||
|
|
||||||
|
def test_protected_bucket_cannot_become_public_across_deployments(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
previous = {
|
||||||
|
"baseline-dist": {
|
||||||
|
"path": "/dist", "access": "protected", "artifact": "old-name",
|
||||||
|
"immutable_prefixes": ["dist/releases"],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
|
||||||
|
deploy.validate_route_migrations(cfg, previous)
|
||||||
|
renamed = copy.deepcopy(cfg)
|
||||||
|
artifact = next(item for item in renamed["artifacts"] if item["name"] == "distributions")
|
||||||
|
artifact["name"] = "downloads"
|
||||||
|
route = next(item for item in renamed["routes"] if item["artifact"] == "distributions")
|
||||||
|
route["artifact"] = "downloads"
|
||||||
|
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
|
||||||
|
deploy.validate_route_migrations(renamed, previous)
|
||||||
|
previous = {"retired-protected-bucket": previous["baseline-dist"]}
|
||||||
|
deploy.validate_route_migrations(cfg, previous)
|
||||||
|
|
||||||
|
def test_protected_split_bucket_cannot_become_legacy_public(self):
|
||||||
|
cfg = normalize_site_config(fixture("legacy-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
previous = {
|
||||||
|
"baseline.fritzlab.net": {
|
||||||
|
"path": "/portal", "access": "protected", "artifact": "portal",
|
||||||
|
"immutable_prefixes": [],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
|
||||||
|
deploy.validate_route_migrations(cfg, previous)
|
||||||
|
|
||||||
|
def test_removed_immutable_rule_preserves_prior_keys(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
|
||||||
|
artifact["cache_rules"] = [
|
||||||
|
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
|
||||||
|
]
|
||||||
|
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
html = Path(tmp)
|
||||||
|
filters = deploy.retired_immutable_filters(artifact, route, html, {
|
||||||
|
"path": "/dist", "access": "public", "artifact": "distributions",
|
||||||
|
"immutable_prefixes": ["dist/releases"],
|
||||||
|
})
|
||||||
|
self.assertEqual(["--exclude", "releases/*"], filters)
|
||||||
|
previous = {
|
||||||
|
"baseline-dist": {
|
||||||
|
"path": "/dist", "access": "public", "artifact": "distributions",
|
||||||
|
"immutable_prefixes": ["dist/releases"],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
next_contracts = deploy.next_route_contracts(cfg, previous)
|
||||||
|
self.assertEqual(
|
||||||
|
["dist/releases"], next_contracts["baseline-dist"]["immutable_prefixes"],
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
["--exclude", "releases/*"],
|
||||||
|
deploy.retired_immutable_filters(
|
||||||
|
artifact, route, html, next_contracts["baseline-dist"],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
(html / "releases").mkdir()
|
||||||
|
(html / "releases" / "replacement.js").write_text("mutable")
|
||||||
|
with self.assertRaisesRegex(RuntimeError, "collides with retired immutable"):
|
||||||
|
deploy.retired_immutable_filters(artifact, route, html, {
|
||||||
|
"path": "/dist", "access": "public", "artifact": "distributions",
|
||||||
|
"immutable_prefixes": ["dist/releases"],
|
||||||
|
})
|
||||||
|
|
||||||
|
def test_removed_route_history_remains_append_only(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
previous = {
|
||||||
|
"retired-bucket": {
|
||||||
|
"path": "/retired", "access": "protected", "artifact": "retired",
|
||||||
|
"immutable_prefixes": ["retired/releases"],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
contracts = deploy.next_route_contracts(cfg, previous)
|
||||||
|
self.assertEqual(previous["retired-bucket"], contracts["retired-bucket"])
|
||||||
|
|
||||||
|
def test_decommission_preserves_history_for_an_unpurged_bucket(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
apps = Path(tmp)
|
||||||
|
site = apps / "sjc001/websites/baseline"
|
||||||
|
site.mkdir(parents=True)
|
||||||
|
history = b'{"schemaVersion":1,"buckets":{}}\n'
|
||||||
|
(site / deploy.HISTORY_FILE).write_bytes(history)
|
||||||
|
(site / "app.yaml").write_text("live\n")
|
||||||
|
with patch.object(deploy, "clone_apps", return_value=apps), patch.object(
|
||||||
|
deploy, "commit_and_push"
|
||||||
|
) as commit:
|
||||||
|
deploy.decommission("baseline", "token", ["baseline-dist"])
|
||||||
|
self.assertEqual(history, (site / deploy.HISTORY_FILE).read_bytes())
|
||||||
|
self.assertFalse((site / "app.yaml").exists())
|
||||||
|
commit.assert_called_once_with(apps, "Decommission baseline", "token")
|
||||||
|
|
||||||
|
def test_later_route_immutable_failure_stops_all_mutable_publication(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
root = Path(tmp)
|
||||||
|
with patch.object(deploy, "validate_publication_environment"), \
|
||||||
|
patch.object(deploy, "validate_artifact_output"), \
|
||||||
|
patch.object(deploy, "clone_apps", return_value=root / "apps"), patch.object(
|
||||||
|
deploy, "publish_route_immutables",
|
||||||
|
side_effect=[None, RuntimeError("immutable failed")],
|
||||||
|
) as immutable_publish, patch.object(deploy, "s3_sync") as mutable_sync, \
|
||||||
|
self.assertRaisesRegex(
|
||||||
|
RuntimeError, "immutable failed"
|
||||||
|
):
|
||||||
|
deploy.deploy_static("baseline", root, root, "token", cfg)
|
||||||
|
|
||||||
|
self.assertEqual(2, immutable_publish.call_count)
|
||||||
|
mutable_sync.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
def test_absent_artifact_is_detected_before_publish(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp, redirect_stderr(io.StringIO()), \
|
||||||
|
self.assertRaises(SystemExit):
|
||||||
|
deploy.validate_artifact_output(Path(tmp), cfg["artifacts"][0])
|
||||||
|
|
||||||
|
def test_absent_cache_prefix_is_detected(self):
|
||||||
|
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
|
||||||
|
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
html = Path(tmp) / artifact["build_dir"]
|
||||||
|
html.mkdir(parents=True)
|
||||||
|
(html / "index.html").write_text("content")
|
||||||
|
with redirect_stderr(io.StringIO()), self.assertRaises(SystemExit):
|
||||||
|
deploy.validate_artifact_output(Path(tmp), artifact)
|
||||||
|
|
||||||
|
|
||||||
|
class ImmutablePublicationTests(unittest.TestCase):
|
||||||
|
CACHE = "public, max-age=31536000, immutable"
|
||||||
|
|
||||||
|
def result(self, returncode, stdout="", stderr=""):
|
||||||
|
return subprocess.CompletedProcess([], returncode, stdout, stderr)
|
||||||
|
|
||||||
|
def key_and_digests(self, source):
|
||||||
|
content_type = "text/javascript"
|
||||||
|
content_digest, publication_digest = deploy._immutable_digests(
|
||||||
|
source, self.CACHE, content_type,
|
||||||
|
)
|
||||||
|
return f"dist/releases/release-{publication_digest}.js", content_digest, publication_digest
|
||||||
|
|
||||||
|
def test_new_key_uses_content_address_and_digest_metadata(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
source = Path(tmp) / "release.js"
|
||||||
|
source.write_text("fixed release")
|
||||||
|
key, content_digest, publication_digest = self.key_and_digests(source)
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def capture(args, _env):
|
||||||
|
calls.append(args)
|
||||||
|
return self.result(1, stderr="404 Not Found") if len(calls) == 1 else self.result(0, "{}")
|
||||||
|
|
||||||
|
with patch.object(deploy, "_aws_capture", side_effect=capture):
|
||||||
|
created = deploy.publish_immutable_file(
|
||||||
|
"http://garage-s3.storage.svc:3900", "dist-bucket",
|
||||||
|
key, source, self.CACHE, {},
|
||||||
|
)
|
||||||
|
self.assertTrue(created)
|
||||||
|
put = calls[1]
|
||||||
|
self.assertEqual("put-object", put[4])
|
||||||
|
self.assertNotIn("--if-none-match", put)
|
||||||
|
self.assertEqual(
|
||||||
|
f"sha256={content_digest},publication-sha256={publication_digest}",
|
||||||
|
put[put.index("--metadata") + 1],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_identical_retry_converges_without_put(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
source = Path(tmp) / "release.js"
|
||||||
|
source.write_text("fixed release")
|
||||||
|
key, content_digest, publication_digest = self.key_and_digests(source)
|
||||||
|
head = json.dumps({
|
||||||
|
"Metadata": {"sha256": content_digest, "publication-sha256": publication_digest},
|
||||||
|
"CacheControl": self.CACHE,
|
||||||
|
"ContentType": "text/javascript",
|
||||||
|
})
|
||||||
|
with patch.object(deploy, "_aws_capture", return_value=self.result(0, head)) as request:
|
||||||
|
created = deploy.publish_immutable_file(
|
||||||
|
"http://garage-s3.storage.svc:3900", "dist-bucket",
|
||||||
|
key, source, self.CACHE, {},
|
||||||
|
)
|
||||||
|
self.assertFalse(created)
|
||||||
|
self.assertEqual(1, request.call_count)
|
||||||
|
|
||||||
|
def test_changed_immutable_key_is_refused(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
source = Path(tmp) / "release.js"
|
||||||
|
source.write_text("changed release")
|
||||||
|
key, _, _ = self.key_and_digests(source)
|
||||||
|
head = json.dumps({"Metadata": {"sha256": "different"}, "CacheControl": self.CACHE})
|
||||||
|
with patch.object(deploy, "_aws_capture", return_value=self.result(0, head)), \
|
||||||
|
self.assertRaisesRegex(RuntimeError, "immutable object differs"):
|
||||||
|
deploy.publish_immutable_file(
|
||||||
|
"http://garage-s3.storage.svc:3900", "dist-bucket",
|
||||||
|
key, source, self.CACHE, {},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_immutable_key_without_publication_digest_is_refused_before_s3(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
source = Path(tmp) / "release.js"
|
||||||
|
source.write_text("fixed release")
|
||||||
|
with patch.object(deploy, "_aws_capture") as request, \
|
||||||
|
self.assertRaisesRegex(RuntimeError, "must contain its one publication SHA-256"):
|
||||||
|
deploy.publish_immutable_file(
|
||||||
|
"http://garage-s3.storage.svc:3900", "dist-bucket",
|
||||||
|
"dist/releases/release.js", source, self.CACHE, {},
|
||||||
|
)
|
||||||
|
request.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user