action/site-publish
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: action/site-publish:artifacts-input
Branches Tags
action/site-publish:main action/site-publish:artifacts-input action/site-publish:feat/split-surface-cors action/site-publish:architect/bug-7acxk8rf0g6b/postmerge-contract-fixes
action/site-publish:v1
..
compare: action/site-publish:70febd3269c08e86f4194a148bba31d6f91597e1
Branches Tags
action/site-publish:main action/site-publish:artifacts-input action/site-publish:feat/split-surface-cors action/site-publish:architect/bug-7acxk8rf0g6b/postmerge-contract-fixes
action/site-publish:v1
1 Commits
artifacts-input .. 70febd3269
Author SHA1 Message Date
Evelyn Chen 70febd3269 fix(site-publish): close split migration boundaries
Test / contract (pull_request) Successful in 7s
Details 
Authored-By: OpenAI (GPT-5) <noreply@openai.com>
 2026-08-29 22:49:45 +00:00
8 changed files with 48 additions and 967 deletions
Expand all files Collapse all files
README.md
+4 -52
View File
@@ -52,7 +52,6 @@ artifacts:
- name: distributions
type: static
content_dir: dist
cors_origins: ['*']
publish:
bucket: baseline-dist
credentials:
@@ -134,16 +133,11 @@ any route's mutable objects change.
Mutable default and override partitions receive their final cache policy before
the matching prefix-scoped stale deletion, so publication never exposes a
provisional cache policy or a pointer to a missing immutable target.
Generated Ingress annotations and `site-publish-history.yaml` retain every
seen bucket's access, prior route, and cumulative bucket-relative immutable key
prefixes, including while an artifact is absent. Removed or renamed rules stay
recorded. When a move places a retired prefix inside the new sync scope, its
Generated Ingress annotations retain each artifact's prior route. When a move
places that retired prefix inside the new sync scope, only its declared
immutable subtrees are excluded; a current-file collision fails publication.
The bucket-keyed history rejects a protected-to-public transition even when the
artifact is renamed; publishing that artifact publicly requires a new bucket.
Legacy single-surface is public for this downgrade check. Removing or renaming
an immutable rule preserves its prior URLs; current mutable content at one of
those paths is rejected instead of replacing it.
The same history rejects a protected-to-public transition that reuses its
protected bucket; publishing that artifact publicly requires a new bucket.
Artifact input directories must be pairwise disjoint after filesystem
resolution. Publication stops before build or upload if one contains another or
@@ -152,12 +146,6 @@ protected input from entering a public artifact through dereference. Split
storage endpoints are pinned to Garage, and each website
authority is derived from its bucket; a site cannot expose an arbitrary backend.
`cors_origins` is reconciled as a bucket policy on every split publication. Values are either `*`
or HTTPS origins; browser access is limited to `GET` and `HEAD`. Omitting the field removes stale
CORS from that bucket. Protected artifacts cannot allow wildcard CORS. All immutable objects and
all bucket CORS policies complete before mutable channels change; if any policy write fails, the
policies already attempted are restored to their pre-publication values.
Each split route gets a bucket-specific `<bucket>.web.sjc001.fritzlab.net`
ExternalName Service annotated to disable pass-host-header and a separate Ingress. Route
Ingresses share the hostname's certificate Secret. The access middleware and
@@ -234,46 +222,10 @@ my-site.fritzlab.net 300 IN CNAME gateway.sjc001.fritzlab.net.
| `garage-admin-token` | legacy aliases only | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
| `username` | no | `ci-bot` | Gitea username |
| `artifacts` | no | every declared artifact | Space- or comma-separated subset of `site.yaml` artifacts to build and publish |
Org secrets in `websites`: `CI_BOT_TOKEN`, `GARAGE_S3_ACCESS_KEY`,
`GARAGE_S3_SECRET_KEY`, `GARAGE_ADMIN_TOKEN`.
### Publishing a subset of artifacts
A repository whose branches ship on different cadences names the ones this run
owns:
```yaml
- uses: https://code.fritzlab.net/action/site-publish@<sha>
with:
token: ${{ secrets.CI_BOT_TOKEN }}
artifacts: catalogue
```
Selection scopes the build and the S3 publication only. Ingresses, Services,
Certificates and the immutable-path history are always rendered from the whole
`site.yaml`, so a scoped run cannot retire another artifact's route or delete
its bucket contents. Credentials are resolved for the selected artifacts alone,
so a workflow need not carry secrets for artifacts it does not publish.
What a scoped run may **not** do is move an artifact it is not publishing. The
generated route contract in `site-publish-history.yaml` is written from the
whole `site.yaml`, and `access` there is a replacement rather than a union — so
recording a change nothing published would turn intent into a fact that
`validate_route_migrations` then refuses to undo. A run whose `site.yaml`
changes an unselected artifact's route path, access or artifact name is refused
before the first bucket is touched, naming both contracts: publish that artifact
in the same run. For the same reason a scoped run cannot introduce an artifact
that has no published history yet.
Two more refusals, both before any bucket changes: an undeclared name, and
`enabled: false` with a selection, because decommissioning is whole-site.
One consequence to know: CORS reconciliation is scoped too, since a scoped run
holds no credential for the other bucket. A `cors_origins:` change lands with
that artifact's next publish, not on the merge that edits `site.yaml`.
## Tools
- **`new-site.sh`** — create a new site: Gitea repo, Garage bucket, web hosting enabled.
action.yaml
-9
View File
@@ -28,13 +28,6 @@ inputs:
description: Gitea username for git operations
required: false
default: ci-bot
artifacts:
# Scopes building and publishing only. Routes, Ingresses and the immutable
# -path history are always rendered from the whole site.yaml, so a scoped
# run never retires another artifact's route.
description: Space- or comma-separated subset of site.yaml artifacts to build and publish (default is every declared artifact)
required: false
default: ''
runs:
using: composite
steps:
@@ -51,7 +44,6 @@ runs:
ACTION_DIR: ${{ github.action_path }}
GITHUB_RUN_NUMBER: ${{ github.run_number }}
CI_BOT_USER: ${{ inputs.username }}
SITE_ARTIFACTS: ${{ inputs.artifacts }}
- name: Deploy
shell: bash
@@ -69,4 +61,3 @@ runs:
GARAGE_ADMIN_ENDPOINT: ${{ inputs.garage-admin-endpoint }}
GARAGE_ADMIN_TOKEN: ${{ inputs.garage-admin-token }}
GITHUB_RUN_NUMBER: ${{ github.run_number }}
SITE_ARTIFACTS: ${{ inputs.artifacts }}
scripts/build.py
+2 -9
View File
@@ -5,14 +5,7 @@ import subprocess
import tempfile
from pathlib import Path
from utils import (
EXCLUDE_FILES,
env,
parse_site_yaml,
run,
selected_artifacts,
validate_artifact_inputs,
)
from utils import EXCLUDE_FILES, env, parse_site_yaml, run, validate_artifact_inputs
def build_artifact(site_dir, artifact):
@@ -73,5 +66,5 @@ def cmd_build():
validate_artifact_inputs(site_dir, cfg)
for artifact in selected_artifacts(cfg):
for artifact in cfg["artifacts"]:
build_artifact(site_dir, artifact)
scripts/deploy.py
+30 -252
View File
@@ -8,7 +8,6 @@ import os
import re
import shutil
import subprocess
import tempfile
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
@@ -25,8 +24,6 @@ from utils import (
parse_site_yaml,
render_templates,
run,
selected_artifacts,
selected_routes,
validate_artifact_inputs,
)
@@ -49,8 +46,8 @@ def validate_artifact_output(site_dir, artifact):
def validate_publication_environment(cfg):
"""Resolve every credential this run needs before the first bucket is changed."""
for artifact in selected_artifacts(cfg):
"""Resolve every declared credential before the first bucket is changed."""
for artifact in cfg["artifacts"]:
env(artifact["credentials"]["access_key_env"])
env(artifact["credentials"]["secret_key_env"])
if cfg["compatibility"] and cfg["aliases"] and not os.environ.get("GARAGE_ADMIN_TOKEN"):
@@ -172,98 +169,6 @@ def publication_aws_env(artifact, credential_env_names=None):
return aws_env
def configure_cors(bucket, origins, endpoint, aws_env):
"""Reconcile read-only browser access without exposing publication credentials."""
if origins is None:
return
config = None
if origins:
config = {
"CORSRules": [{
"AllowedOrigins": origins,
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 3600,
}],
}
set_cors_configuration(bucket, config, endpoint, aws_env)
def set_cors_configuration(bucket, config, endpoint, aws_env):
"""Apply an exact bucket CORS configuration, or remove it when absent."""
if config is None:
run([
"aws", "--endpoint-url", endpoint, "s3api", "delete-bucket-cors",
"--bucket", bucket,
], env=aws_env)
return
with tempfile.NamedTemporaryFile("w", suffix=".json", encoding="utf-8") as handle:
json.dump(config, handle)
handle.flush()
run([
"aws", "--endpoint-url", endpoint, "s3api", "put-bucket-cors",
"--bucket", bucket, "--cors-configuration", f"file://{handle.name}",
], env=aws_env)
def get_cors_configuration(bucket, endpoint, aws_env):
"""Read the exact bucket CORS configuration for rollback."""
result = _aws_capture([
"aws", "--endpoint-url", endpoint, "s3api", "get-bucket-cors",
"--bucket", bucket, "--output", "json",
], aws_env)
if result.returncode == 0:
try:
config = json.loads(result.stdout)
except json.JSONDecodeError as error:
raise RuntimeError(f"get-bucket-cors returned invalid JSON for {bucket}") from error
if not isinstance(config, dict) or not isinstance(config.get("CORSRules"), list):
raise RuntimeError(f"get-bucket-cors returned an invalid policy for {bucket}")
return config
error = f"{result.stdout}\n{result.stderr}"
if "NoSuchCORSConfiguration" in error:
return None
raise RuntimeError(f"get-bucket-cors failed for {bucket}: {error.strip()}")
def reconcile_artifact_cors(artifacts, credential_env_names=None):
"""Reconcile all policies, restoring the prior set if any write fails."""
snapshots = []
for artifact in artifacts:
if artifact["cors_origins"] is None:
continue
aws_env = publication_aws_env(artifact, credential_env_names)
snapshots.append((
artifact,
aws_env,
get_cors_configuration(artifact["bucket"], artifact["s3_endpoint"], aws_env),
))
attempted = []
try:
for artifact, aws_env, previous in snapshots:
attempted.append((artifact, aws_env, previous))
configure_cors(
artifact["bucket"], artifact["cors_origins"], artifact["s3_endpoint"], aws_env,
)
except Exception as error:
rollback_errors = []
for artifact, aws_env, previous in reversed(attempted):
try:
set_cors_configuration(
artifact["bucket"], previous, artifact["s3_endpoint"], aws_env,
)
except Exception as rollback_error:
rollback_errors.append(f"{artifact['bucket']}: {rollback_error}")
if rollback_errors:
raise RuntimeError(
f"CORS reconciliation failed ({error}); rollback also failed for "
f"{'; '.join(rollback_errors)}"
) from error
raise
def publish_route_immutables(artifact, route, site_dir, credential_env_names=None):
"""Publish one route's immutable partitions during the global preflight."""
html_dir = site_dir / artifact["build_dir"]
@@ -273,38 +178,22 @@ def publish_route_immutables(artifact, route, site_dir, credential_env_names=Non
publish_immutable_rule(artifact, route, rule, html_dir, aws_env)
def immutable_key_prefixes(artifact, route):
"""Return immutable partitions as bucket-relative key prefixes."""
route_prefix = route["path"].strip("/")
return [
"/".join(part for part in (route_prefix, rule["path"]) if part)
for rule in artifact["cache_rules"] if _is_immutable(rule)
]
def retained_immutable_paths(artifact, route, previous_contract):
"""Carry all bucket history forward so later route moves cannot delete it."""
previous_paths = previous_contract["immutable_paths"] if previous_contract else []
return sorted(set(previous_paths) | set(immutable_key_prefixes(artifact, route)))
def retired_immutable_filters(artifact, route, html_dir, previous_contract):
"""Protect historical immutable keys that fall inside the current sync scope."""
if not previous_contract:
def retired_immutable_filters(artifact, route, html_dir, previous_path):
"""Protect immutable keys only when an old route falls inside the new scope."""
if not previous_path or previous_path == route["path"]:
return []
current_prefix = route["path"].strip("/")
filters = []
current_immutable = set(immutable_key_prefixes(artifact, route))
for immutable_path in previous_contract["immutable_paths"]:
if immutable_path in current_immutable:
continue
previous_prefix = previous_path.strip("/")
if current_prefix:
marker = f"{current_prefix}/"
if not immutable_path.startswith(marker):
if not previous_prefix.startswith(marker):
return []
previous_prefix = previous_prefix[len(marker):]
filters = []
for rule in artifact["cache_rules"]:
if not _is_immutable(rule):
continue
retired_path = immutable_path[len(marker):]
else:
retired_path = immutable_path
retired_path = "/".join(part for part in (previous_prefix, rule["path"]) if part)
collision = html_dir / retired_path
if collision.exists() and any(path.is_file() for path in collision.rglob("*")):
raise RuntimeError(
@@ -314,7 +203,7 @@ def retired_immutable_filters(artifact, route, html_dir, previous_contract):
return filters
def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_contract=None):
def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_path=None):
endpoint = artifact["s3_endpoint"]
html_dir = site_dir / artifact["build_dir"]
aws_env = publication_aws_env(artifact, credential_env_names)
@@ -342,7 +231,7 @@ def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_contr
# so a fresh upload always carries the right MIME type.
specific_paths = [rule["path"] for rule in artifact["cache_rules"] if rule["path"]]
default_filters = [arg for path in specific_paths for arg in ("--exclude", f"{path}/*")]
retired_filters = retired_immutable_filters(artifact, route, html_dir, previous_contract)
retired_filters = retired_immutable_filters(artifact, route, html_dir, previous_path)
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
"--recursive", "--only-show-errors", "--cache-control", default_cache,
*default_filters, *retired_filters, *exclude_args], env=aws_env)
@@ -414,32 +303,16 @@ def ensure_bucket_aliases(site_name, aliases, admin_token):
raise
def render_site_manifests(
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts=None,
):
def render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg):
"""Always re-render manifests from current site.yaml. Templates own
domain + aliases, so changes propagate without manual edits."""
manifests_dir.mkdir(parents=True, exist_ok=True)
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
routes = []
previous_contracts = previous_contracts or {}
next_contracts = {bucket: dict(contract)
for bucket, contract in previous_contracts.items()}
for route in cfg["routes"]:
artifact = artifact_by_name[route["artifact"]]
resource_name = k8s_name(site_name) if cfg["compatibility"] else f"{k8s_name(site_name)}-{route['name']}"
previous = previous_contracts.get(artifact["bucket"])
immutable_paths = retained_immutable_paths(artifact, route, previous)
next_contracts[artifact["bucket"]] = {
"path": route["path"],
"access": "public" if route["access"] == "legacy" else route["access"],
"artifact": route["artifact"],
"immutable_paths": immutable_paths,
}
routes.append({
**route, "resource_name": resource_name, "artifact_config": artifact,
"immutable_paths_json": json.dumps(immutable_paths, separators=(",", ":")),
})
routes.append({**route, "resource_name": resource_name, "artifact_config": artifact})
template_vars = {
"site": site_name,
"site_k8s": k8s_name(site_name),
@@ -450,44 +323,10 @@ def render_site_manifests(
"routes": routes,
}
render_templates(action_dir, template_vars, app_dir, manifests_dir)
if not cfg["compatibility"] or previous_contracts:
(app_dir / "site-publish-history.yaml").write_text(yaml.safe_dump(
{"version": 1, "buckets": next_contracts}, sort_keys=True,
))
def _validate_route_contract(bucket, contract, path):
expected = {"path", "access", "artifact", "immutable_paths"}
if (not isinstance(bucket, str) or not bucket or not isinstance(contract, dict)
or set(contract) != expected
or contract.get("access") not in {"public", "protected"}
or not isinstance(contract.get("path"), str)
or not contract["path"].startswith("/")
or not isinstance(contract.get("artifact"), str) or not contract["artifact"]
or not isinstance(contract.get("immutable_paths"), list)
or any(not isinstance(item, str) or not item or item.startswith("/")
or any(part in {"", ".", ".."} for part in item.split("/"))
for item in contract["immutable_paths"])):
raise RuntimeError(f"invalid site-publish route history in {path}")
return {
"path": contract["path"], "access": contract["access"],
"artifact": contract["artifact"],
"immutable_paths": sorted(set(contract["immutable_paths"])),
}
def previous_route_contracts(app_dir):
"""Read bucket-keyed route history from generated Ingresses."""
history_path = app_dir / "site-publish-history.yaml"
if history_path.exists():
document = yaml.safe_load(history_path.read_text())
if (not isinstance(document, dict) or set(document) != {"version", "buckets"}
or document["version"] != 1 or not isinstance(document["buckets"], dict)):
raise RuntimeError(f"invalid site-publish route history in {history_path}")
return {
bucket: _validate_route_contract(bucket, contract, history_path)
for bucket, contract in document["buckets"].items()
}
"""Read artifact route, access, and bucket history from generated Ingresses."""
contracts = {}
manifests = app_dir / "manifests"
if not manifests.exists():
@@ -498,120 +337,61 @@ def previous_route_contracts(app_dir):
artifact = annotations.get("site-publish.fritzlab.net/artifact")
access = annotations.get("site-publish.fritzlab.net/access")
bucket = annotations.get("site-publish.fritzlab.net/bucket")
immutable_paths_json = annotations.get("site-publish.fritzlab.net/immutable-paths")
route_path = annotations.get("site-publish.fritzlab.net/route-path")
values = (artifact, access, bucket, immutable_paths_json, route_path)
values = (artifact, access, bucket, route_path)
if all(value is None for value in values):
continue
if (not all(isinstance(value, str) for value in values)
or access not in {"public", "protected"} or not route_path.startswith("/")):
raise RuntimeError(f"invalid site-publish route history in {path}")
try:
immutable_paths = json.loads(immutable_paths_json)
except json.JSONDecodeError as exc:
raise RuntimeError(f"invalid site-publish route history in {path}") from exc
if not isinstance(immutable_paths, list) or any(not isinstance(item, str) for item in immutable_paths):
raise RuntimeError(f"invalid site-publish route history in {path}")
if bucket in contracts:
raise RuntimeError(f"duplicate site-publish route history for bucket {bucket}")
contracts[bucket] = _validate_route_contract(bucket, {
"path": route_path, "access": access, "artifact": artifact,
"immutable_paths": immutable_paths,
}, path)
if artifact in contracts:
raise RuntimeError(f"duplicate site-publish route history for artifact {artifact}")
contracts[artifact] = {"path": route_path, "access": access, "bucket": bucket}
return contracts
def validate_route_migrations(cfg, previous_contracts):
artifacts = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
for route in cfg["routes"]:
previous = previous_contracts.get(route["artifact"])
artifact = artifacts[route["artifact"]]
previous = previous_contracts.get(artifact["bucket"])
if (previous and previous["access"] == "protected"
and route["access"] in {"public", "legacy"}
):
if (previous and previous["access"] == "protected" and route["access"] == "public"
and previous["bucket"] == artifact["bucket"]):
raise RuntimeError(
f"artifact {route['artifact']} cannot become public while reusing protected "
f"bucket {artifact['bucket']}"
)
def validate_scoped_history(cfg, previous_contracts):
"""A scoped run may not record a route contract it did not publish.
render_site_manifests advances the stored contract for every route in
site.yaml, and `access` is overwritten rather than unioned the way
immutable_paths is. Without this, a catalogue-only publish could write a
protected access for the distributions bucket that nothing published, and
validate_route_migrations would then refuse to put that bucket back —
unpublished intent turned into an irreversible fact.
"""
chosen = set(cfg["selected"])
if len(chosen) == len(cfg["artifacts"]):
return
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
for route in cfg["routes"]:
if route["artifact"] in chosen:
continue
artifact = artifact_by_name[route["artifact"]]
previous = previous_contracts.get(artifact["bucket"])
if previous is None:
raise RuntimeError(
f"a scoped run cannot introduce artifact {route['artifact']}; "
f"publish it in the same run"
)
current = {
"path": route["path"],
"access": "public" if route["access"] == "legacy" else route["access"],
"artifact": route["artifact"],
}
recorded = {key: previous[key] for key in current}
if recorded != current:
raise RuntimeError(
f"a scoped run may not change unselected artifact {route['artifact']}'s route "
f"contract ({recorded} -> {current}); publish it in the same run"
)
def deploy_static(site_name, site_dir, action_dir, token, cfg):
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
credential_env_names = {
name for artifact in cfg["artifacts"] for name in artifact["credentials"].values()
}
# Publication is scoped to the selected artifacts; the rendered route
# contract is not. Manifests and immutable-path history stay whole, so a
# partial publish can never retire another artifact's route or bucket.
publishing = selected_routes(cfg)
validate_publication_environment(cfg)
for artifact in selected_artifacts(cfg):
for artifact in cfg["artifacts"]:
validate_artifact_output(site_dir, artifact)
apps_dir = clone_apps(token)
app_dir = apps_dir / "sjc001" / "websites" / site_name
manifests_dir = app_dir / "manifests"
previous_contracts = previous_route_contracts(app_dir)
validate_route_migrations(cfg, previous_contracts)
validate_scoped_history(cfg, previous_contracts)
# Complete immutable work across the whole publication before any route's
# mutable pointers can change. Partial immutable success is safe; mixing a
# new route with an old route after a later immutable failure is not.
for route in publishing:
for route in cfg["routes"]:
publish_route_immutables(
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
)
# Reconcile every browser-read policy before publishing mutable content.
# A CORS failure therefore cannot leave a new channel pointing at a release
# whose cross-origin assets browsers cannot consume.
reconcile_artifact_cors(selected_artifacts(cfg), credential_env_names)
for route in publishing:
for route in cfg["routes"]:
s3_sync(
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
previous_contracts.get(artifact_by_name[route["artifact"]]["bucket"]),
(previous_contracts.get(route["artifact"]) or {}).get("path"),
)
if cfg["compatibility"]:
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
render_site_manifests(
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts,
)
render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg)
commit_and_push(apps_dir, f"Deploy {site_name}", token)
@@ -640,8 +420,6 @@ def cmd_deploy():
cfg = parse_site_yaml(site_dir)
if not cfg["enabled"]:
if len(cfg["selected"]) != len(cfg["artifacts"]):
die("a disabled site decommissions whole; drop the artifacts selection")
print("Site disabled — running decommission...")
decommission(site_name, token, [artifact["bucket"] for artifact in cfg["artifacts"]])
return
scripts/utils.py
+1 -95
View File
@@ -1,6 +1,5 @@
"""Shared utilities for the site-publish action."""
import ipaddress
import os
import re
import shutil
@@ -105,55 +104,6 @@ def _strings(value, label):
return values
def _cors_origins(value, label):
origins = _list(value, label)
if any(not isinstance(item, str) or not item for item in origins):
raise ConfigError(f"{label} must be a list of non-empty strings")
canonical = []
for origin in origins:
if origin == "*":
canonical.append(origin)
continue
parsed = urlparse(origin)
try:
port = parsed.port
except ValueError:
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins") from None
if parsed.scheme != "https" or not parsed.hostname or parsed.path or parsed.params or (
parsed.query or parsed.fragment or parsed.username or parsed.password
):
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
try:
address = ipaddress.ip_address(parsed.hostname)
except ValueError:
try:
hostname = parsed.hostname.encode("idna").decode("ascii")
except UnicodeError:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
) from None
try:
_hostname(hostname, f"{label} hostname")
except ConfigError:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
) from None
else:
if getattr(address, "scope_id", None) is not None:
raise ConfigError(
f"{label} must contain '*' or canonical HTTPS origins"
)
hostname = f"[{address.compressed}]" if address.version == 6 else address.compressed
canonical.append(f"https://{hostname}{f':{port}' if port not in (None, 443) else ''}")
if len(canonical) != len(set(canonical)):
raise ConfigError(f"{label} must not contain duplicate canonical origins")
if "*" in canonical and len(canonical) != 1:
raise ConfigError(f"{label} wildcard must be the only origin")
if origins != canonical:
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
return origins
def _hostname(value, label):
if not isinstance(value, str) or len(value) > 253 or value.endswith("."):
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
@@ -265,7 +215,6 @@ def _legacy_config(raw, site_name):
"website_authority": "garage-s3.storage.svc.k8s.sjc001.fritzlab.net",
"credentials": {"access_key_env": "AWS_ACCESS_KEY_ID", "secret_key_env": "AWS_SECRET_ACCESS_KEY"},
"cache_rules": [{"path": "", "cache_control": DEFAULT_CACHE_CONTROL}],
"cors_origins": None,
}
return {
"version": 1,
@@ -274,7 +223,6 @@ def _legacy_config(raw, site_name):
"aliases": _aliases(raw.get("aliases"), raw["domain"]),
"enabled": raw.get("enabled", True),
"artifacts": [artifact],
"selected": ["site"],
"routes": [{
"name": "site", "path": "/", "artifact": "site", "access": "legacy",
"access_middleware": None,
@@ -286,11 +234,7 @@ def _legacy_config(raw, site_name):
def _artifact(item, index):
label = f"artifacts[{index}]"
item = _mapping(item, label)
_known_keys(
item,
{"name", "type", "content_dir", "tidy", "excludes", "publish", "cache", "cors_origins"},
label,
)
_known_keys(item, {"name", "type", "content_dir", "tidy", "excludes", "publish", "cache"}, label)
name = item.get("name")
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
raise ConfigError(f"{label}.name must be a DNS label")
@@ -359,10 +303,6 @@ def _artifact(item, index):
"website_authority": authority,
"credentials": normalized_credentials,
"cache_rules": cache_rules,
"cors_origins": (
_cors_origins(item["cors_origins"], f"{label}.cors_origins")
if "cors_origins" in item else []
),
}
@@ -449,42 +389,11 @@ def _validate_multi(cfg):
raise ConfigError(f"protected route {route['name']} cannot use shared-cache max-age")
if route["access"] == "public" and "private" in directives:
raise ConfigError(f"public route {route['name']} cannot use private cache policy")
if route["access"] == "protected" and "*" in artifact["cors_origins"]:
raise ConfigError(
f"protected route {route['name']} cannot allow wildcard CORS"
)
root = next(route for route in routes if route["path"] == "/")
if any(route["access"] == "protected" for route in routes) and root["access"] == "public":
raise ConfigError("a public '/' catch-all would expose unmatched protected content")
def _artifact_selection(cfg, requested):
"""Resolve the subset of declared artifacts this run publishes."""
declared = [artifact["name"] for artifact in cfg["artifacts"]]
if not requested or not requested.strip():
return declared
names = [part for part in re.split(r"[,\s]+", requested.strip()) if part]
unknown = sorted({name for name in names if name not in declared})
if unknown:
raise ConfigError(
"artifacts selects undeclared artifact(s) " + ", ".join(unknown)
+ "; site.yaml declares " + ", ".join(declared)
)
return [name for name in declared if name in set(names)]
def selected_artifacts(cfg):
"""Artifacts this run builds and publishes, in declaration order."""
chosen = set(cfg["selected"])
return [artifact for artifact in cfg["artifacts"] if artifact["name"] in chosen]
def selected_routes(cfg):
"""Routes whose artifact this run publishes, in route order."""
chosen = set(cfg["selected"])
return [route for route in cfg["routes"] if route["artifact"] in chosen]
def normalize_site_config(raw, site_name):
raw = _mapping(raw, "site.yaml")
domain = _hostname(raw.get("domain"), "domain")
@@ -510,7 +419,6 @@ def normalize_site_config(raw, site_name):
"routes": sorted(routes, key=lambda item: (-len(item["path"]), item["path"], item["name"])),
}
_validate_multi(cfg)
cfg["selected"] = [artifact["name"] for artifact in cfg["artifacts"]]
for route in cfg["routes"]:
if len(f"{k8s_name(site_name)}-{route['name']}") > 63:
raise ConfigError(f"route {route['name']} makes the generated Service name exceed 63 characters")
@@ -563,14 +471,12 @@ def parse_site_yaml(site_dir, site_name=None):
try:
with open(path) as stream:
cfg = normalize_site_config(yaml.safe_load(stream), site_name)
cfg["selected"] = _artifact_selection(cfg, os.environ.get("SITE_ARTIFACTS"))
except (ConfigError, yaml.YAMLError) as exc:
die(str(exc))
print("Site config:")
print(f" domain: {cfg['domain']}")
print(f" contract: {cfg['compatibility'] or 'split-surface-v2'}")
print(f" artifacts: {[item['name'] for item in cfg['artifacts']]}")
print(f" publishing: {cfg['selected']}")
print(f" routes: {[(item['path'], item['access']) for item in cfg['routes']]}")
return cfg
templates/ingress.yaml.j2
+3 -6
View File
@@ -4,13 +4,10 @@ metadata:
name: {{ route.resource_name }}
namespace: {{ namespace }}
annotations:
{%- if not compatibility %}
site-publish.fritzlab.net/artifact: {{ route.artifact | tojson }}
site-publish.fritzlab.net/access: {{ route.access | tojson }}
site-publish.fritzlab.net/bucket: {{ route.artifact_config.bucket | tojson }}
site-publish.fritzlab.net/immutable-paths: {{ route.immutable_paths_json | tojson }}
site-publish.fritzlab.net/artifact: {{ route.artifact }}
site-publish.fritzlab.net/access: {{ route.access }}
site-publish.fritzlab.net/bucket: {{ route.artifact_config.bucket }}
site-publish.fritzlab.net/route-path: {{ route.path | tojson }}
{%- endif %}
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% if route.access_middleware %},{{ route.access_middleware }}@file{% endif %}{% for m in route.middlewares %},{{ m }}@file{% endfor %}
spec:
ingressClassName: traefik
tests/fixtures/split-site.yaml
Vendored
-1
View File
@@ -15,7 +15,6 @@ artifacts:
- name: distributions
type: static
content_dir: dist
cors_origins: ['*']
publish:
bucket: baseline-dist
credentials:
tests/test_contract.py
+7 -542
View File
@@ -24,12 +24,7 @@ sys.path.insert(0, str(ROOT / "scripts"))
import deploy
import build
import utils
from utils import (
ConfigError,
normalize_site_config,
parse_site_yaml,
validate_artifact_inputs,
)
from utils import ConfigError, normalize_site_config, validate_artifact_inputs
def fixture(name):
@@ -118,69 +113,6 @@ class ConfigContractTests(unittest.TestCase):
self.assertEqual(["/dist", "/"], [route["path"] for route in cfg["routes"]])
credentials = {artifact["name"]: artifact["credentials"] for artifact in cfg["artifacts"]}
self.assertNotEqual(credentials["distributions"], credentials["portal"])
distributions = next(
artifact for artifact in cfg["artifacts"] if artifact["name"] == "distributions"
)
self.assertEqual(["*"], distributions["cors_origins"])
def test_cors_origins_are_https_origins_or_wildcard(self):
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["http://consumer.example"]
),
"must contain '\\*' or canonical HTTPS origins",
)
for origin in (
"https://consumer example",
"https://Consumer.example",
"https://consumer.example:443",
"https://consumer.example/",
"https://consumer.example:invalid",
"https://[fe80::1%eth0]",
"https://[fe80::1%25eth0]",
):
with self.subTest(origin=origin):
self.assert_invalid(
lambda raw, origin=origin: raw["artifacts"][1].__setitem__(
"cors_origins", [origin]
),
"must contain '\\*' or canonical HTTPS origins",
)
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["https://consumer.example", "https://consumer.example:443"]
),
"must not contain duplicate canonical origins",
)
self.assert_invalid(
lambda raw: raw["artifacts"][1].__setitem__(
"cors_origins", ["*", "https://consumer.example"]
),
"wildcard must be the only origin",
)
raw = copy.deepcopy(self.raw)
raw["artifacts"][1]["cors_origins"] = ["https://[2602:817:3000::1]:8443"]
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
self.assertEqual(
["https://[2602:817:3000::1]:8443"],
next(a for a in cfg["artifacts"] if a["name"] == "distributions")["cors_origins"],
)
def test_declared_cors_origins_must_be_a_list(self):
for value in (None, False, 0, "", {}):
with self.subTest(value=value):
self.assert_invalid(
lambda raw, value=value: raw["artifacts"][1].__setitem__(
"cors_origins", value
),
"cors_origins must be a list",
)
def test_protected_route_rejects_wildcard_cors(self):
self.assert_invalid(
lambda raw: raw["artifacts"][0].__setitem__("cors_origins", ["*"]),
"protected route portal cannot allow wildcard CORS",
)
def test_equivalent_route_paths_are_ambiguous(self):
self.assert_invalid(lambda raw: raw["routes"].append({
@@ -199,7 +131,6 @@ class ConfigContractTests(unittest.TestCase):
raw["routes"][1]["access"] = {
"mode": "protected", "middleware": "authentik-forwardauth"
}
raw["artifacts"][1]["cors_origins"] = []
raw["artifacts"][0]["cache"]["rules"][0]["cache_control"] = (
"public, max-age=0, must-revalidate"
)
@@ -332,7 +263,6 @@ class GenerationTests(unittest.TestCase):
"app.yaml", "manifests/certificate.yaml", "manifests/ingress-distributions.yaml",
"manifests/ingress-portal.yaml", "manifests/kustomization.yaml",
"manifests/service-distributions.yaml", "manifests/service-portal.yaml",
"site-publish-history.yaml",
}, set(files))
for content in files.values():
self.assertIsNotNone(yaml.safe_load(content))
@@ -343,17 +273,6 @@ class GenerationTests(unittest.TestCase):
self.assertNotIn("passhostheader", files["manifests/ingress-portal.yaml"])
self.assertIn("baseline-dist.web.sjc001.fritzlab.net", files["manifests/service-distributions.yaml"])
def test_yaml_ambiguous_artifact_name_stays_a_string_annotation(self):
raw = fixture("split-site.yaml")
raw["artifacts"][0]["name"] = "yes"
raw["routes"][0]["artifact"] = "yes"
tmp, _, files = self.render(raw)
self.addCleanup(tmp.cleanup)
ingress = yaml.safe_load(files["manifests/ingress-portal.yaml"])
self.assertEqual(
"yes", ingress["metadata"]["annotations"]["site-publish.fritzlab.net/artifact"],
)
def test_generation_is_deterministic_when_input_lists_are_reversed(self):
raw = fixture("split-site.yaml")
first_tmp, _, first = self.render(raw)
@@ -377,14 +296,12 @@ class GenerationTests(unittest.TestCase):
self.assertFalse(stale.exists())
def test_legacy_names_and_garage_s3_target_are_preserved(self):
tmp, app_dir, files = self.render(fixture("legacy-site.yaml"))
tmp, _, files = self.render(fixture("legacy-site.yaml"))
self.addCleanup(tmp.cleanup)
self.assertIn("manifests/service.yaml", files)
self.assertIn("manifests/ingress.yaml", files)
self.assertIn("garage-s3.storage.svc.k8s.sjc001.fritzlab.net", files["manifests/service.yaml"])
self.assertNotIn("passhostheader", files["manifests/ingress.yaml"])
self.assertNotIn("site-publish.fritzlab.net", files["manifests/ingress.yaml"])
self.assertEqual({}, deploy.previous_route_contracts(app_dir))
class BuildTests(unittest.TestCase):
@@ -409,285 +326,7 @@ class BuildTests(unittest.TestCase):
)
class ArtifactSelectionTests(unittest.TestCase):
"""The `artifacts` input scopes publication, never the route contract."""
def parse(self, selection):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "site.yaml").write_text(
(ROOT / "tests" / "fixtures" / "split-site.yaml").read_text()
)
environment = {"SITE_REPO": "fritzlab/baseline"}
if selection is not None:
environment["SITE_ARTIFACTS"] = selection
with patch.dict(os.environ, environment, clear=False), \
redirect_stdout(io.StringIO()):
if selection is None:
os.environ.pop("SITE_ARTIFACTS", None)
return parse_site_yaml(root)
def test_absent_selection_publishes_every_declared_artifact(self):
self.assertEqual(["distributions", "portal"], self.parse(None)["selected"])
self.assertEqual(["distributions", "portal"], self.parse(" ")["selected"])
def test_selection_accepts_comma_and_whitespace_separated_names(self):
self.assertEqual(["portal"], self.parse("portal")["selected"])
self.assertEqual(
["distributions", "portal"], self.parse("portal, distributions")["selected"]
)
self.assertEqual(
["distributions", "portal"], self.parse("portal distributions")["selected"]
)
def test_undeclared_selection_fails_before_any_publication(self):
stderr = io.StringIO()
with redirect_stderr(stderr), self.assertRaises(SystemExit):
self.parse("catalogue")
self.assertIn("undeclared artifact", stderr.getvalue())
self.assertIn("catalogue", stderr.getvalue())
def test_build_skips_unselected_artifacts(self):
raw = fixture("split-site.yaml")
for artifact in raw["artifacts"]:
artifact["tidy"] = False
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
cfg["selected"] = ["portal"]
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "portal" / "build").mkdir(parents=True)
(root / "portal" / "build" / "index.html").write_text("portal")
built = []
with patch.object(build, "build_artifact",
side_effect=lambda _root, artifact: built.append(artifact["name"])), \
patch.dict(os.environ, {"SITE_DIR": str(root)}, clear=False), \
patch.object(build, "parse_site_yaml", return_value=cfg), \
redirect_stdout(io.StringIO()):
build.cmd_build()
self.assertEqual(["portal"], built)
def deploy_selection(self, selection, environment):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
cfg["selected"] = selection
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
root = Path(tmp.name)
apps_dir = root / "apps"
app_dir = apps_dir / "sjc001" / "websites" / "baseline.fritzlab.net"
app_dir.mkdir(parents=True)
# A scoped run refuses an artifact it has no published history for, so
# the fixture starts from the whole publication this one narrows.
(app_dir / "site-publish-history.yaml").write_text(yaml.safe_dump({
"version": 1,
"buckets": {
artifact["bucket"]: {
"path": route["path"],
"access": "public" if route["access"] == "legacy" else route["access"],
"artifact": route["artifact"],
"immutable_paths": deploy.immutable_key_prefixes(artifact, route),
}
for route in cfg["routes"]
for artifact in [next(item for item in cfg["artifacts"]
if item["name"] == route["artifact"])]
},
}, sort_keys=True))
for artifact in cfg["artifacts"]:
if artifact["name"] not in set(selection):
continue
html = root / artifact["build_dir"]
html.mkdir(parents=True)
(html / "index.html").write_text(artifact["name"])
for rule in artifact["cache_rules"]:
if rule["path"]:
(html / rule["path"]).mkdir(parents=True, exist_ok=True)
(html / rule["path"] / "keep.json").write_text("{}")
synced, immutables, cors = [], [], []
with patch.dict(os.environ, environment, clear=False), \
patch.object(deploy, "clone_apps", return_value=apps_dir), \
patch.object(deploy, "commit_and_push"), \
patch.object(deploy, "publish_route_immutables",
side_effect=lambda artifact, *_a, **_k: immutables.append(artifact["name"])), \
patch.object(deploy, "reconcile_artifact_cors",
side_effect=lambda artifacts, *_a: cors.extend(item["name"] for item in artifacts)), \
patch.object(deploy, "s3_sync",
side_effect=lambda artifact, *_a, **_k: synced.append(artifact["name"])), \
redirect_stdout(io.StringIO()):
deploy.deploy_static("baseline.fritzlab.net", root, ROOT, "token", cfg)
files = {path.relative_to(app_dir).as_posix(): path.read_text()
for path in sorted(app_dir.rglob("*.yaml"))}
return {"synced": synced, "immutables": immutables, "cors": cors, "files": files}
def test_selected_publication_leaves_the_whole_route_contract_rendered(self):
# Only the portal credentials exist: a scoped run must not demand the
# secrets of an artifact it is not publishing.
environment = {"PORTAL_S3_ACCESS_KEY": "portal-key",
"PORTAL_S3_SECRET_KEY": "portal-secret"}
for name in ("DIST_S3_ACCESS_KEY", "DIST_S3_SECRET_KEY"):
os.environ.pop(name, None)
result = self.deploy_selection(["portal"], environment)
self.assertEqual(["portal"], result["synced"])
self.assertEqual(["portal"], result["immutables"])
self.assertEqual(["portal"], result["cors"])
self.assertIn("manifests/ingress-distributions.yaml", result["files"])
self.assertIn("manifests/service-distributions.yaml", result["files"])
history = yaml.safe_load(result["files"]["site-publish-history.yaml"])
self.assertEqual({"baseline-dist", "baseline-portal"}, set(history["buckets"]))
self.assertEqual(
["dist/releases"], history["buckets"]["baseline-dist"]["immutable_paths"]
)
def test_scoped_run_refuses_to_record_a_contract_it_did_not_publish(self):
"""The reviewer's scenario: distributions goes protected in site.yaml while
a catalogue-only run publishes. Recording that access would make the
bucket unreturnable through validate_route_migrations."""
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
published = {
"baseline-portal": {
"path": "/", "access": "protected", "artifact": "portal", "immutable_paths": [],
},
"baseline-dist": {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
},
}
# A whole publish agrees with what is recorded, scoped or not.
deploy.validate_scoped_history(cfg, published)
cfg["selected"] = ["portal"]
deploy.validate_scoped_history(cfg, published)
moved = copy.deepcopy(cfg)
route = next(item for item in moved["routes"] if item["artifact"] == "distributions")
route["access"] = "protected"
route["access_middleware"] = "authentik-forwardauth"
with self.assertRaises(RuntimeError) as refused:
deploy.validate_scoped_history(moved, published)
self.assertIn("may not change unselected artifact distributions", str(refused.exception))
# The same change published in the same run is allowed to proceed.
moved["selected"] = ["distributions", "portal"]
deploy.validate_scoped_history(moved, published)
def test_scoped_run_refuses_an_artifact_with_no_published_history(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
cfg["selected"] = ["portal"]
with self.assertRaises(RuntimeError) as refused:
deploy.validate_scoped_history(cfg, {})
self.assertIn("cannot introduce artifact distributions", str(refused.exception))
def test_scoped_deploy_runs_the_history_guard_before_touching_a_bucket(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
cfg["selected"] = ["portal"]
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
root = Path(tmp.name)
apps_dir = root / "apps"
app_dir = apps_dir / "sjc001" / "websites" / "baseline.fritzlab.net"
app_dir.mkdir(parents=True)
html = root / next(
item for item in cfg["artifacts"] if item["name"] == "portal"
)["build_dir"]
html.mkdir(parents=True)
(html / "index.html").write_text("portal")
with patch.dict(os.environ, {
"PORTAL_S3_ACCESS_KEY": "portal-key", "PORTAL_S3_SECRET_KEY": "portal-secret",
}, clear=False), \
patch.object(deploy, "clone_apps", return_value=apps_dir), \
patch.object(deploy, "commit_and_push"), \
patch.object(deploy, "publish_route_immutables") as immutables, \
patch.object(deploy, "reconcile_artifact_cors") as cors, \
patch.object(deploy, "s3_sync") as sync, \
redirect_stdout(io.StringIO()), \
self.assertRaises(RuntimeError):
deploy.deploy_static("baseline.fritzlab.net", root, ROOT, "token", cfg)
immutables.assert_not_called()
cors.assert_not_called()
sync.assert_not_called()
def test_disabled_site_refuses_a_partial_selection(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
cfg["enabled"] = False
cfg["selected"] = ["portal"]
with patch.object(deploy, "parse_site_yaml", return_value=cfg), \
patch.dict(os.environ, {
"SITE_REPO": "fritzlab/baseline.fritzlab.net", "SITE_DIR": ".",
"ACTION_DIR": str(ROOT), "CI_BOT_TOKEN": "token",
}, clear=False), \
patch.object(deploy, "decommission") as decommission, \
redirect_stderr(io.StringIO()), redirect_stdout(io.StringIO()), \
self.assertRaises(SystemExit):
deploy.cmd_deploy()
decommission.assert_not_called()
class PublishingTests(unittest.TestCase):
def test_cors_policy_is_reconciled_as_read_only_browser_access(self):
captured = {}
def capture(command, **_kwargs):
config_path = command[command.index("--cors-configuration") + 1]
captured["command"] = command
captured["config"] = json.loads(Path(config_path.removeprefix("file://")).read_text())
with patch.object(deploy, "run", side_effect=capture):
deploy.configure_cors(
"baseline-dist", ["*"], "http://garage-s3.storage.svc:3900", {}
)
self.assertIn("put-bucket-cors", captured["command"])
self.assertEqual(["GET", "HEAD"], captured["config"]["CORSRules"][0]["AllowedMethods"])
self.assertEqual(["*"], captured["config"]["CORSRules"][0]["AllowedOrigins"])
def test_empty_cors_policy_removes_stale_bucket_cors(self):
with patch.object(deploy, "run") as request:
deploy.configure_cors(
"baseline-catalogue", [], "http://garage-s3.storage.svc:3900", {}
)
self.assertIn("delete-bucket-cors", request.call_args.args[0])
def test_cors_reconciliation_restores_prior_policies_on_failure(self):
artifacts = [
{
"bucket": "first", "cors_origins": ["https://new.example"],
"s3_endpoint": "http://garage-s3.storage.svc:3900", "credentials": {},
},
{
"bucket": "second", "cors_origins": [],
"s3_endpoint": "http://garage-s3.storage.svc:3900", "credentials": {},
},
]
prior = [
{"CORSRules": [{"AllowedOrigins": ["https://old.example"]}]},
None,
]
writes = []
def write(bucket, config, *_args):
writes.append((bucket, config))
if bucket == "second" and len(writes) == 2:
raise RuntimeError("write failed")
with patch.object(deploy, "publication_aws_env", return_value={}), patch.object(
deploy, "get_cors_configuration", side_effect=prior,
) as read, patch.object(deploy, "set_cors_configuration", side_effect=write), \
self.assertRaisesRegex(RuntimeError, "write failed"):
deploy.reconcile_artifact_cors(artifacts)
self.assertEqual(2, read.call_count)
self.assertEqual(
[
("first", {"CORSRules": [{
"AllowedOrigins": ["https://new.example"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag"],
"MaxAgeSeconds": 3600,
}]}),
("second", None),
("second", None),
("first", prior[0]),
],
writes,
)
def test_apps_clone_never_places_token_in_argv_or_log(self):
calls = []
secret = "clone-secret-must-not-appear"
@@ -819,10 +458,7 @@ class PublishingTests(unittest.TestCase):
}, clear=False), patch.object(
deploy, "run", side_effect=lambda command, **_: commands.append(command)
):
deploy.s3_sync(artifact, route, root, previous_contract={
"path": "/foo", "access": "public", "artifact": "distributions",
"immutable_paths": ["foo/releases"],
})
deploy.s3_sync(artifact, route, root, previous_path="/foo")
rendered = [" ".join(command) for command in commands]
self.assertTrue(all("foo/releases/*" in command for command in rendered[:2]))
self.assertTrue(all("*/releases/*" not in command for command in rendered))
@@ -830,159 +466,15 @@ class PublishingTests(unittest.TestCase):
def test_protected_bucket_cannot_become_public_across_deployments(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
previous = {
"baseline-dist": {
"path": "/dist", "access": "protected", "artifact": "old-name",
"immutable_paths": ["dist/releases"],
"distributions": {
"path": "/dist", "access": "protected", "bucket": "baseline-dist",
}
}
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(cfg, previous)
renamed = copy.deepcopy(cfg)
artifact = next(item for item in renamed["artifacts"] if item["name"] == "distributions")
artifact["name"] = "downloads"
route = next(item for item in renamed["routes"] if item["artifact"] == "distributions")
route["artifact"] = "downloads"
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(renamed, previous)
previous = {"retired-protected-bucket": previous["baseline-dist"]}
previous["distributions"]["bucket"] = "retired-protected-bucket"
deploy.validate_route_migrations(cfg, previous)
def test_protected_split_bucket_cannot_become_legacy_public(self):
cfg = normalize_site_config(fixture("legacy-site.yaml"), "baseline.fritzlab.net")
previous = {
"baseline.fritzlab.net": {
"path": "/portal", "access": "protected", "artifact": "portal",
"immutable_paths": [],
}
}
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(cfg, previous)
def test_removed_immutable_rule_preserves_prior_keys(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
with tempfile.TemporaryDirectory() as tmp:
html = Path(tmp)
filters = deploy.retired_immutable_filters(artifact, route, html, {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
})
self.assertEqual(["--exclude", "releases/*"], filters)
(html / "releases").mkdir()
(html / "releases" / "replacement.js").write_text("mutable")
with self.assertRaisesRegex(RuntimeError, "collides with retired immutable"):
deploy.retired_immutable_filters(artifact, route, html, {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
})
def test_removed_immutable_rule_remains_in_next_manifest_history(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
previous = {
"baseline-dist": {
"path": "/dist", "access": "public", "artifact": "distributions",
"immutable_paths": ["dist/releases"],
}
}
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp) / "app"
manifests = app_dir / "manifests"
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, previous,
)
first = deploy.previous_route_contracts(app_dir)
self.assertEqual(["dist/releases"], first["baseline-dist"]["immutable_paths"])
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, first,
)
second = deploy.previous_route_contracts(app_dir)
self.assertEqual(first, second)
def test_protected_split_bucket_survives_absence_and_blocks_legacy(self):
raw = fixture("split-site.yaml")
raw["artifacts"] = [
item for item in raw["artifacts"] if item["name"] == "distributions"
]
raw["routes"] = [
item for item in raw["routes"] if item["artifact"] == "distributions"
]
raw["routes"][0]["path"] = "/"
cfg = normalize_site_config(raw, "baseline.fritzlab.net")
previous = {
"baseline.fritzlab.net": {
"path": "/", "access": "protected", "artifact": "portal",
"immutable_paths": [],
}
}
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp) / "app"
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, app_dir / "manifests", cfg, previous,
)
retained = deploy.previous_route_contracts(app_dir)
self.assertEqual(
previous["baseline.fritzlab.net"], retained["baseline.fritzlab.net"],
)
legacy = normalize_site_config(
fixture("legacy-site.yaml"), "baseline.fritzlab.net",
)
with self.assertRaisesRegex(RuntimeError, "cannot become public while reusing protected"):
deploy.validate_route_migrations(legacy, retained)
def test_malformed_persistent_history_fails_closed(self):
with tempfile.TemporaryDirectory() as tmp:
app_dir = Path(tmp)
(app_dir / "site-publish-history.yaml").write_text(
"version: 1\nbuckets:\n bucket:\n path: /\n"
" access: public\n artifact: site\n"
" immutable_paths: [../releases]\n"
)
with self.assertRaisesRegex(RuntimeError, "invalid site-publish route history"):
deploy.previous_route_contracts(app_dir)
def test_retired_immutable_history_survives_a_route_move(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
artifact = next(item for item in cfg["artifacts"] if item["name"] == "distributions")
artifact["cache_rules"] = [
rule for rule in artifact["cache_rules"] if rule["path"] != "releases"
]
route = next(item for item in cfg["routes"] if item["artifact"] == "distributions")
route["path"] = "/"
previous = {
"baseline-dist": {
"path": "/foo", "access": "public", "artifact": "distributions",
"immutable_paths": ["foo/releases"],
}
}
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
app_dir = root / "app"
manifests = app_dir / "manifests"
app_dir.mkdir()
deploy.render_site_manifests(
"baseline.fritzlab.net", ROOT, app_dir, manifests, cfg, previous,
)
following = deploy.previous_route_contracts(app_dir)
self.assertEqual(["foo/releases"], following["baseline-dist"]["immutable_paths"])
html = root / "html"
html.mkdir()
first_filters = deploy.retired_immutable_filters(
artifact, route, html, previous["baseline-dist"],
)
following_filters = deploy.retired_immutable_filters(
artifact, route, html, following["baseline-dist"],
)
self.assertEqual(["--exclude", "foo/releases/*"], first_filters)
self.assertEqual(first_filters, following_filters)
def test_later_route_immutable_failure_stops_all_mutable_publication(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
with tempfile.TemporaryDirectory() as tmp:
@@ -992,42 +484,15 @@ class PublishingTests(unittest.TestCase):
patch.object(deploy, "clone_apps", return_value=root / "apps"), patch.object(
deploy, "publish_route_immutables",
side_effect=[None, RuntimeError("immutable failed")],
) as immutable_publish, patch.object(
deploy, "reconcile_artifact_cors"
) as cors_reconcile, patch.object(deploy, "s3_sync") as mutable_sync, \
) as immutable_publish, patch.object(deploy, "s3_sync") as mutable_sync, \
self.assertRaisesRegex(
RuntimeError, "immutable failed"
):
deploy.deploy_static("baseline", root, root, "token", cfg)
self.assertEqual(2, immutable_publish.call_count)
cors_reconcile.assert_not_called()
mutable_sync.assert_not_called()
def test_all_cors_policies_complete_before_mutable_publication(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")
events = []
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
apps = root / "apps"
apps.mkdir()
with patch.object(deploy, "validate_publication_environment"), patch.object(
deploy, "validate_artifact_output"
), patch.object(
deploy, "publish_route_immutables", side_effect=lambda *_args: events.append("immutable")
), patch.object(
deploy, "reconcile_artifact_cors", side_effect=lambda *_args: events.append("cors")
), patch.object(
deploy, "s3_sync", side_effect=lambda *_args: events.append("mutable")
), patch.object(deploy, "clone_apps", return_value=apps), patch.object(
deploy, "render_site_manifests"
), patch.object(deploy, "commit_and_push"):
deploy.deploy_static("baseline", root, root, "token", cfg)
self.assertEqual(
["immutable", "immutable", "cors", "mutable", "mutable"], events
)
def test_absent_artifact_is_detected_before_publish(self):
cfg = normalize_site_config(fixture("split-site.yaml"), "baseline.fritzlab.net")