Compare commits
18
Commits
7667ae540e
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
910341391f | ||
|
|
304095436c | ||
|
|
b14f6a856b | ||
|
|
3dfee64335 | ||
|
|
9287e4861a | ||
|
|
5f4325706b | ||
|
|
310ae6a29d | ||
|
|
173f0a3a6d | ||
|
|
deccc4e177 | ||
|
|
9b0a8c4fd4 | ||
|
|
898816db16 | ||
|
|
892b6e6441 | ||
|
|
85a0b41380 | ||
|
|
19fb4e43ab | ||
|
|
7b824a61ca | ||
|
|
fb2e440bbb | ||
|
|
5261b9b99f | ||
|
|
5c2630b972 |
@@ -52,6 +52,7 @@ artifacts:
|
|||||||
- name: distributions
|
- name: distributions
|
||||||
type: static
|
type: static
|
||||||
content_dir: dist
|
content_dir: dist
|
||||||
|
cors_origins: ['*']
|
||||||
publish:
|
publish:
|
||||||
bucket: baseline-dist
|
bucket: baseline-dist
|
||||||
credentials:
|
credentials:
|
||||||
@@ -92,8 +93,10 @@ routes:
|
|||||||
```
|
```
|
||||||
|
|
||||||
The caller supplies each declared credential name as an environment variable
|
The caller supplies each declared credential name as an environment variable
|
||||||
on the action step. Credential values are passed to `aws` only through its
|
on the action step. Names must be matched `<NAME>_S3_ACCESS_KEY` and
|
||||||
environment and never appear in a logged command or process argument.
|
`<NAME>_S3_SECRET_KEY` pairs; arbitrary environment variables cannot become
|
||||||
|
publication credentials. Values pass to `aws` only through its environment and
|
||||||
|
never appear in a logged command or process argument.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
- uses: https://code.fritzlab.net/action/site-publish@v1
|
- uses: https://code.fritzlab.net/action/site-publish@v1
|
||||||
@@ -120,6 +123,40 @@ revalidation, or `no-store` plus a positive max-age) are rejected. Protected
|
|||||||
artifacts require `private` or `no-store` and cannot emit `public`.
|
artifacts require `private` or `no-store` and cannot emit `public`.
|
||||||
Metadata restamping transfers each artifact once even on a no-op publication;
|
Metadata restamping transfers each artifact once even on a no-op publication;
|
||||||
that is the cost of making policy changes effective on unchanged Garage objects.
|
that is the cost of making policy changes effective on unchanged Garage objects.
|
||||||
|
An immutable cache path is excluded from sync and deletion. Every object key in
|
||||||
|
that path must contain exactly one full publication SHA-256, calculated over its
|
||||||
|
cache policy, content type, and bytes. That content address makes concurrent
|
||||||
|
writes identical even though Garage v2.2.0 has no conditional destination
|
||||||
|
write. An identical retry converges; a changed object, missing digest metadata,
|
||||||
|
wrong address, or nested policy under that immutable prefix fails publication.
|
||||||
|
Every immutable target across every artifact is validated and published before
|
||||||
|
any route's mutable objects change.
|
||||||
|
Mutable default and override partitions receive their final cache policy before
|
||||||
|
the matching prefix-scoped stale deletion, so publication never exposes a
|
||||||
|
provisional cache policy or a pointer to a missing immutable target.
|
||||||
|
Generated Ingress annotations and `site-publish-history.yaml` retain every
|
||||||
|
seen bucket's access, prior route, and cumulative bucket-relative immutable key
|
||||||
|
prefixes, including while an artifact is absent. Removed or renamed rules stay
|
||||||
|
recorded. When a move places a retired prefix inside the new sync scope, its
|
||||||
|
immutable subtrees are excluded; a current-file collision fails publication.
|
||||||
|
The bucket-keyed history rejects a protected-to-public transition even when the
|
||||||
|
artifact is renamed; publishing that artifact publicly requires a new bucket.
|
||||||
|
Legacy single-surface is public for this downgrade check. Removing or renaming
|
||||||
|
an immutable rule preserves its prior URLs; current mutable content at one of
|
||||||
|
those paths is rejected instead of replacing it.
|
||||||
|
|
||||||
|
Artifact input directories must be pairwise disjoint after filesystem
|
||||||
|
resolution. Publication stops before build or upload if one contains another or
|
||||||
|
escapes the repository. Symlinked roots, components, and descendants are also rejected, preventing
|
||||||
|
protected input from entering a public artifact through dereference. Split
|
||||||
|
storage endpoints are pinned to Garage, and each website
|
||||||
|
authority is derived from its bucket; a site cannot expose an arbitrary backend.
|
||||||
|
|
||||||
|
`cors_origins` is reconciled as a bucket policy on every split publication. Values are either `*`
|
||||||
|
or HTTPS origins; browser access is limited to `GET` and `HEAD`. Omitting the field removes stale
|
||||||
|
CORS from that bucket. Protected artifacts cannot allow wildcard CORS. All immutable objects and
|
||||||
|
all bucket CORS policies complete before mutable channels change; if any policy write fails, the
|
||||||
|
policies already attempted are restored to their pre-publication values.
|
||||||
|
|
||||||
Each split route gets a bucket-specific `<bucket>.web.sjc001.fritzlab.net`
|
Each split route gets a bucket-specific `<bucket>.web.sjc001.fritzlab.net`
|
||||||
ExternalName Service annotated to disable pass-host-header and a separate Ingress. Route
|
ExternalName Service annotated to disable pass-host-header and a separate Ingress. Route
|
||||||
@@ -197,10 +234,46 @@ my-site.fritzlab.net 300 IN CNAME gateway.sjc001.fritzlab.net.
|
|||||||
| `garage-admin-token` | legacy aliases only | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
|
| `garage-admin-token` | legacy aliases only | | Garage admin API token (`admin-token` from `garage-rpc-secret` in `storage` ns) |
|
||||||
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
|
| `garage-admin-endpoint` | no | `http://garage.storage.svc:3903` | Garage admin API endpoint |
|
||||||
| `username` | no | `ci-bot` | Gitea username |
|
| `username` | no | `ci-bot` | Gitea username |
|
||||||
|
| `artifacts` | no | every declared artifact | Space- or comma-separated subset of `site.yaml` artifacts to build and publish |
|
||||||
|
|
||||||
Org secrets in `websites`: `CI_BOT_TOKEN`, `GARAGE_S3_ACCESS_KEY`,
|
Org secrets in `websites`: `CI_BOT_TOKEN`, `GARAGE_S3_ACCESS_KEY`,
|
||||||
`GARAGE_S3_SECRET_KEY`, `GARAGE_ADMIN_TOKEN`.
|
`GARAGE_S3_SECRET_KEY`, `GARAGE_ADMIN_TOKEN`.
|
||||||
|
|
||||||
|
### Publishing a subset of artifacts
|
||||||
|
|
||||||
|
A repository whose branches ship on different cadences names the ones this run
|
||||||
|
owns:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- uses: https://code.fritzlab.net/action/site-publish@<sha>
|
||||||
|
with:
|
||||||
|
token: ${{ secrets.CI_BOT_TOKEN }}
|
||||||
|
artifacts: catalogue
|
||||||
|
```
|
||||||
|
|
||||||
|
Selection scopes the build and the S3 publication only. Ingresses, Services,
|
||||||
|
Certificates and the immutable-path history are always rendered from the whole
|
||||||
|
`site.yaml`, so a scoped run cannot retire another artifact's route or delete
|
||||||
|
its bucket contents. Credentials are resolved for the selected artifacts alone,
|
||||||
|
so a workflow need not carry secrets for artifacts it does not publish.
|
||||||
|
|
||||||
|
What a scoped run may **not** do is move an artifact it is not publishing. The
|
||||||
|
generated route contract in `site-publish-history.yaml` is written from the
|
||||||
|
whole `site.yaml`, and `access` there is a replacement rather than a union — so
|
||||||
|
recording a change nothing published would turn intent into a fact that
|
||||||
|
`validate_route_migrations` then refuses to undo. A run whose `site.yaml`
|
||||||
|
changes an unselected artifact's route path, access or artifact name is refused
|
||||||
|
before the first bucket is touched, naming both contracts: publish that artifact
|
||||||
|
in the same run. For the same reason a scoped run cannot introduce an artifact
|
||||||
|
that has no published history yet.
|
||||||
|
|
||||||
|
Two more refusals, both before any bucket changes: an undeclared name, and
|
||||||
|
`enabled: false` with a selection, because decommissioning is whole-site.
|
||||||
|
|
||||||
|
One consequence to know: CORS reconciliation is scoped too, since a scoped run
|
||||||
|
holds no credential for the other bucket. A `cors_origins:` change lands with
|
||||||
|
that artifact's next publish, not on the merge that edits `site.yaml`.
|
||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
|
|
||||||
- **`new-site.sh`** — create a new site: Gitea repo, Garage bucket, web hosting enabled.
|
- **`new-site.sh`** — create a new site: Gitea repo, Garage bucket, web hosting enabled.
|
||||||
|
|||||||
@@ -28,6 +28,13 @@ inputs:
|
|||||||
description: Gitea username for git operations
|
description: Gitea username for git operations
|
||||||
required: false
|
required: false
|
||||||
default: ci-bot
|
default: ci-bot
|
||||||
|
artifacts:
|
||||||
|
# Scopes building and publishing only. Routes, Ingresses and the immutable
|
||||||
|
# -path history are always rendered from the whole site.yaml, so a scoped
|
||||||
|
# run never retires another artifact's route.
|
||||||
|
description: Space- or comma-separated subset of site.yaml artifacts to build and publish (default is every declared artifact)
|
||||||
|
required: false
|
||||||
|
default: ''
|
||||||
runs:
|
runs:
|
||||||
using: composite
|
using: composite
|
||||||
steps:
|
steps:
|
||||||
@@ -44,6 +51,7 @@ runs:
|
|||||||
ACTION_DIR: ${{ github.action_path }}
|
ACTION_DIR: ${{ github.action_path }}
|
||||||
GITHUB_RUN_NUMBER: ${{ github.run_number }}
|
GITHUB_RUN_NUMBER: ${{ github.run_number }}
|
||||||
CI_BOT_USER: ${{ inputs.username }}
|
CI_BOT_USER: ${{ inputs.username }}
|
||||||
|
SITE_ARTIFACTS: ${{ inputs.artifacts }}
|
||||||
|
|
||||||
- name: Deploy
|
- name: Deploy
|
||||||
shell: bash
|
shell: bash
|
||||||
@@ -61,3 +69,4 @@ runs:
|
|||||||
GARAGE_ADMIN_ENDPOINT: ${{ inputs.garage-admin-endpoint }}
|
GARAGE_ADMIN_ENDPOINT: ${{ inputs.garage-admin-endpoint }}
|
||||||
GARAGE_ADMIN_TOKEN: ${{ inputs.garage-admin-token }}
|
GARAGE_ADMIN_TOKEN: ${{ inputs.garage-admin-token }}
|
||||||
GITHUB_RUN_NUMBER: ${{ github.run_number }}
|
GITHUB_RUN_NUMBER: ${{ github.run_number }}
|
||||||
|
SITE_ARTIFACTS: ${{ inputs.artifacts }}
|
||||||
|
|||||||
+11
-2
@@ -5,7 +5,14 @@ import subprocess
|
|||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from utils import EXCLUDE_FILES, env, parse_site_yaml, run
|
from utils import (
|
||||||
|
EXCLUDE_FILES,
|
||||||
|
env,
|
||||||
|
parse_site_yaml,
|
||||||
|
run,
|
||||||
|
selected_artifacts,
|
||||||
|
validate_artifact_inputs,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_artifact(site_dir, artifact):
|
def build_artifact(site_dir, artifact):
|
||||||
@@ -64,5 +71,7 @@ def cmd_build():
|
|||||||
print("Site disabled — skipping build")
|
print("Site disabled — skipping build")
|
||||||
return
|
return
|
||||||
|
|
||||||
for artifact in cfg["artifacts"]:
|
validate_artifact_inputs(site_dir, cfg)
|
||||||
|
|
||||||
|
for artifact in selected_artifacts(cfg):
|
||||||
build_artifact(site_dir, artifact)
|
build_artifact(site_dir, artifact)
|
||||||
|
|||||||
+451
-43
@@ -1,13 +1,20 @@
|
|||||||
"""Deploy phase — S3 sync, manifest rendering, alias reconcile."""
|
"""Deploy phase — S3 sync, manifest rendering, alias reconcile."""
|
||||||
|
|
||||||
|
import fnmatch
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import mimetypes
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from urllib.error import HTTPError, URLError
|
from urllib.error import HTTPError, URLError
|
||||||
from urllib.request import Request, urlopen
|
from urllib.request import Request, urlopen
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
from utils import (
|
from utils import (
|
||||||
NAMESPACE,
|
NAMESPACE,
|
||||||
clone_apps,
|
clone_apps,
|
||||||
@@ -18,6 +25,9 @@ from utils import (
|
|||||||
parse_site_yaml,
|
parse_site_yaml,
|
||||||
render_templates,
|
render_templates,
|
||||||
run,
|
run,
|
||||||
|
selected_artifacts,
|
||||||
|
selected_routes,
|
||||||
|
validate_artifact_inputs,
|
||||||
)
|
)
|
||||||
|
|
||||||
GARAGE_ADMIN_ENDPOINT = os.environ.get(
|
GARAGE_ADMIN_ENDPOINT = os.environ.get(
|
||||||
@@ -39,17 +49,113 @@ def validate_artifact_output(site_dir, artifact):
|
|||||||
|
|
||||||
|
|
||||||
def validate_publication_environment(cfg):
|
def validate_publication_environment(cfg):
|
||||||
"""Resolve every declared credential before the first bucket is changed."""
|
"""Resolve every credential this run needs before the first bucket is changed."""
|
||||||
for artifact in cfg["artifacts"]:
|
for artifact in selected_artifacts(cfg):
|
||||||
env(artifact["credentials"]["access_key_env"])
|
env(artifact["credentials"]["access_key_env"])
|
||||||
env(artifact["credentials"]["secret_key_env"])
|
env(artifact["credentials"]["secret_key_env"])
|
||||||
if cfg["compatibility"] and cfg["aliases"] and not os.environ.get("GARAGE_ADMIN_TOKEN"):
|
if cfg["compatibility"] and cfg["aliases"] and not os.environ.get("GARAGE_ADMIN_TOKEN"):
|
||||||
die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
|
die("GARAGE_ADMIN_TOKEN is required when aliases are declared")
|
||||||
|
|
||||||
|
|
||||||
def s3_sync(artifact, route, site_dir, credential_env_names=None):
|
def _is_immutable(rule):
|
||||||
endpoint = artifact["s3_endpoint"]
|
return "immutable" in {
|
||||||
html_dir = site_dir / artifact["build_dir"]
|
part.strip().lower().split("=", 1)[0]
|
||||||
|
for part in rule["cache_control"].split(",")
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _aws_capture(args, aws_env):
|
||||||
|
"""Run a non-streaming AWS request without exposing environment credentials."""
|
||||||
|
print(f" $ {' '.join(str(part) for part in args)}")
|
||||||
|
return subprocess.run(args, env=aws_env, text=True, capture_output=True, check=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _immutable_head(endpoint, bucket, key, aws_env):
|
||||||
|
args = ["aws", "--endpoint-url", endpoint, "s3api", "head-object",
|
||||||
|
"--bucket", bucket, "--key", key, "--output", "json"]
|
||||||
|
result = _aws_capture(args, aws_env)
|
||||||
|
if result.returncode == 0:
|
||||||
|
return json.loads(result.stdout)
|
||||||
|
error = f"{result.stdout}\n{result.stderr}"
|
||||||
|
if any(marker in error for marker in ("404", "Not Found", "NoSuchKey")):
|
||||||
|
return None
|
||||||
|
raise RuntimeError(f"head-object failed for s3://{bucket}/{key}: {error.strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def _immutable_digests(source, cache_control, content_type):
|
||||||
|
with source.open("rb") as stream:
|
||||||
|
content_digest = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||||
|
publication = hashlib.sha256()
|
||||||
|
publication.update(cache_control.encode())
|
||||||
|
publication.update(b"\0")
|
||||||
|
publication.update(content_type.encode())
|
||||||
|
publication.update(b"\0")
|
||||||
|
with source.open("rb") as stream:
|
||||||
|
for block in iter(lambda: stream.read(1024 * 1024), b""):
|
||||||
|
publication.update(block)
|
||||||
|
return content_digest, publication.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _same_immutable_object(info, content_digest, publication_digest, cache_control, content_type):
|
||||||
|
metadata = {key.lower(): value for key, value in (info.get("Metadata") or {}).items()}
|
||||||
|
return (
|
||||||
|
metadata.get("sha256") == content_digest
|
||||||
|
and metadata.get("publication-sha256") == publication_digest
|
||||||
|
and info.get("CacheControl") == cache_control
|
||||||
|
and info.get("ContentType") == content_type
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def publish_immutable_file(endpoint, bucket, key, source, cache_control, aws_env):
|
||||||
|
"""Publish a content-addressed key; identical retries converge."""
|
||||||
|
content_type = mimetypes.guess_type(source.name)[0] or "application/octet-stream"
|
||||||
|
content_digest, publication_digest = _immutable_digests(source, cache_control, content_type)
|
||||||
|
address_digests = re.findall(r"(?<![0-9a-f])([0-9a-f]{64})(?![0-9a-f])", key.lower())
|
||||||
|
if address_digests != [publication_digest]:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"immutable key must contain its one publication SHA-256 {publication_digest}: "
|
||||||
|
f"s3://{bucket}/{key}"
|
||||||
|
)
|
||||||
|
existing = _immutable_head(endpoint, bucket, key, aws_env)
|
||||||
|
if existing is not None:
|
||||||
|
if _same_immutable_object(
|
||||||
|
existing, content_digest, publication_digest, cache_control, content_type,
|
||||||
|
):
|
||||||
|
print(f" Immutable object already matches: s3://{bucket}/{key}")
|
||||||
|
return False
|
||||||
|
raise RuntimeError(f"immutable object differs or lacks publisher digest: s3://{bucket}/{key}")
|
||||||
|
args = ["aws", "--endpoint-url", endpoint, "s3api", "put-object",
|
||||||
|
"--bucket", bucket, "--key", key, "--body", str(source),
|
||||||
|
"--content-type", content_type, "--cache-control", cache_control,
|
||||||
|
"--metadata", f"sha256={content_digest},publication-sha256={publication_digest}"]
|
||||||
|
result = _aws_capture(args, aws_env)
|
||||||
|
if result.returncode == 0:
|
||||||
|
return True
|
||||||
|
error = f"{result.stdout}\n{result.stderr}"
|
||||||
|
raise RuntimeError(f"put-object failed for s3://{bucket}/{key}: {error.strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def publish_immutable_rule(artifact, route, rule, html_dir, aws_env):
|
||||||
|
"""Publish one immutable cache partition without overwrite or deletion."""
|
||||||
|
rule_root = html_dir / rule["path"]
|
||||||
|
child_paths = [candidate["path"] for candidate in artifact["cache_rules"]
|
||||||
|
if candidate["path"].startswith(f"{rule['path'].rstrip('/')}/")]
|
||||||
|
object_prefix = route["path"].strip("/")
|
||||||
|
for source in sorted(path for path in rule_root.rglob("*") if path.is_file()):
|
||||||
|
relative = source.relative_to(html_dir).as_posix()
|
||||||
|
if any(relative == child or relative.startswith(f"{child}/") for child in child_paths):
|
||||||
|
continue
|
||||||
|
if any(fnmatch.fnmatch(relative, pattern) for pattern in artifact["excludes"]):
|
||||||
|
continue
|
||||||
|
key = "/".join(part for part in (object_prefix, relative) if part)
|
||||||
|
publish_immutable_file(
|
||||||
|
artifact["s3_endpoint"], artifact["bucket"], key, source,
|
||||||
|
rule["cache_control"], aws_env,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def publication_aws_env(artifact, credential_env_names=None):
|
||||||
|
"""Build the route-scoped AWS environment without leaking other credentials."""
|
||||||
access_key = env(artifact["credentials"]["access_key_env"])
|
access_key = env(artifact["credentials"]["access_key_env"])
|
||||||
secret_key = env(artifact["credentials"]["secret_key_env"])
|
secret_key = env(artifact["credentials"]["secret_key_env"])
|
||||||
aws_env = os.environ.copy()
|
aws_env = os.environ.copy()
|
||||||
@@ -63,11 +169,160 @@ def s3_sync(artifact, route, site_dir, credential_env_names=None):
|
|||||||
"AWS_SECRET_ACCESS_KEY": secret_key,
|
"AWS_SECRET_ACCESS_KEY": secret_key,
|
||||||
"AWS_DEFAULT_REGION": os.environ.get("AWS_DEFAULT_REGION", "sjc001"),
|
"AWS_DEFAULT_REGION": os.environ.get("AWS_DEFAULT_REGION", "sjc001"),
|
||||||
})
|
})
|
||||||
|
return aws_env
|
||||||
|
|
||||||
|
|
||||||
|
def configure_cors(bucket, origins, endpoint, aws_env):
|
||||||
|
"""Reconcile read-only browser access without exposing publication credentials."""
|
||||||
|
if origins is None:
|
||||||
|
return
|
||||||
|
config = None
|
||||||
|
if origins:
|
||||||
|
config = {
|
||||||
|
"CORSRules": [{
|
||||||
|
"AllowedOrigins": origins,
|
||||||
|
"AllowedMethods": ["GET", "HEAD"],
|
||||||
|
"AllowedHeaders": ["*"],
|
||||||
|
"ExposeHeaders": ["ETag"],
|
||||||
|
"MaxAgeSeconds": 3600,
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
set_cors_configuration(bucket, config, endpoint, aws_env)
|
||||||
|
|
||||||
|
|
||||||
|
def set_cors_configuration(bucket, config, endpoint, aws_env):
|
||||||
|
"""Apply an exact bucket CORS configuration, or remove it when absent."""
|
||||||
|
if config is None:
|
||||||
|
run([
|
||||||
|
"aws", "--endpoint-url", endpoint, "s3api", "delete-bucket-cors",
|
||||||
|
"--bucket", bucket,
|
||||||
|
], env=aws_env)
|
||||||
|
return
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".json", encoding="utf-8") as handle:
|
||||||
|
json.dump(config, handle)
|
||||||
|
handle.flush()
|
||||||
|
run([
|
||||||
|
"aws", "--endpoint-url", endpoint, "s3api", "put-bucket-cors",
|
||||||
|
"--bucket", bucket, "--cors-configuration", f"file://{handle.name}",
|
||||||
|
], env=aws_env)
|
||||||
|
|
||||||
|
|
||||||
|
def get_cors_configuration(bucket, endpoint, aws_env):
|
||||||
|
"""Read the exact bucket CORS configuration for rollback."""
|
||||||
|
result = _aws_capture([
|
||||||
|
"aws", "--endpoint-url", endpoint, "s3api", "get-bucket-cors",
|
||||||
|
"--bucket", bucket, "--output", "json",
|
||||||
|
], aws_env)
|
||||||
|
if result.returncode == 0:
|
||||||
|
try:
|
||||||
|
config = json.loads(result.stdout)
|
||||||
|
except json.JSONDecodeError as error:
|
||||||
|
raise RuntimeError(f"get-bucket-cors returned invalid JSON for {bucket}") from error
|
||||||
|
if not isinstance(config, dict) or not isinstance(config.get("CORSRules"), list):
|
||||||
|
raise RuntimeError(f"get-bucket-cors returned an invalid policy for {bucket}")
|
||||||
|
return config
|
||||||
|
error = f"{result.stdout}\n{result.stderr}"
|
||||||
|
if "NoSuchCORSConfiguration" in error:
|
||||||
|
return None
|
||||||
|
raise RuntimeError(f"get-bucket-cors failed for {bucket}: {error.strip()}")
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile_artifact_cors(artifacts, credential_env_names=None):
|
||||||
|
"""Reconcile all policies, restoring the prior set if any write fails."""
|
||||||
|
snapshots = []
|
||||||
|
for artifact in artifacts:
|
||||||
|
if artifact["cors_origins"] is None:
|
||||||
|
continue
|
||||||
|
aws_env = publication_aws_env(artifact, credential_env_names)
|
||||||
|
snapshots.append((
|
||||||
|
artifact,
|
||||||
|
aws_env,
|
||||||
|
get_cors_configuration(artifact["bucket"], artifact["s3_endpoint"], aws_env),
|
||||||
|
))
|
||||||
|
|
||||||
|
attempted = []
|
||||||
|
try:
|
||||||
|
for artifact, aws_env, previous in snapshots:
|
||||||
|
attempted.append((artifact, aws_env, previous))
|
||||||
|
configure_cors(
|
||||||
|
artifact["bucket"], artifact["cors_origins"], artifact["s3_endpoint"], aws_env,
|
||||||
|
)
|
||||||
|
except Exception as error:
|
||||||
|
rollback_errors = []
|
||||||
|
for artifact, aws_env, previous in reversed(attempted):
|
||||||
|
try:
|
||||||
|
set_cors_configuration(
|
||||||
|
artifact["bucket"], previous, artifact["s3_endpoint"], aws_env,
|
||||||
|
)
|
||||||
|
except Exception as rollback_error:
|
||||||
|
rollback_errors.append(f"{artifact['bucket']}: {rollback_error}")
|
||||||
|
if rollback_errors:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"CORS reconciliation failed ({error}); rollback also failed for "
|
||||||
|
f"{'; '.join(rollback_errors)}"
|
||||||
|
) from error
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def publish_route_immutables(artifact, route, site_dir, credential_env_names=None):
|
||||||
|
"""Publish one route's immutable partitions during the global preflight."""
|
||||||
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
|
aws_env = publication_aws_env(artifact, credential_env_names)
|
||||||
|
for rule in artifact["cache_rules"]:
|
||||||
|
if _is_immutable(rule):
|
||||||
|
publish_immutable_rule(artifact, route, rule, html_dir, aws_env)
|
||||||
|
|
||||||
|
|
||||||
|
def immutable_key_prefixes(artifact, route):
|
||||||
|
"""Return immutable partitions as bucket-relative key prefixes."""
|
||||||
|
route_prefix = route["path"].strip("/")
|
||||||
|
return [
|
||||||
|
"/".join(part for part in (route_prefix, rule["path"]) if part)
|
||||||
|
for rule in artifact["cache_rules"] if _is_immutable(rule)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def retained_immutable_paths(artifact, route, previous_contract):
|
||||||
|
"""Carry all bucket history forward so later route moves cannot delete it."""
|
||||||
|
previous_paths = previous_contract["immutable_paths"] if previous_contract else []
|
||||||
|
return sorted(set(previous_paths) | set(immutable_key_prefixes(artifact, route)))
|
||||||
|
|
||||||
|
|
||||||
|
def retired_immutable_filters(artifact, route, html_dir, previous_contract):
|
||||||
|
"""Protect historical immutable keys that fall inside the current sync scope."""
|
||||||
|
if not previous_contract:
|
||||||
|
return []
|
||||||
|
current_prefix = route["path"].strip("/")
|
||||||
|
filters = []
|
||||||
|
current_immutable = set(immutable_key_prefixes(artifact, route))
|
||||||
|
for immutable_path in previous_contract["immutable_paths"]:
|
||||||
|
if immutable_path in current_immutable:
|
||||||
|
continue
|
||||||
|
if current_prefix:
|
||||||
|
marker = f"{current_prefix}/"
|
||||||
|
if not immutable_path.startswith(marker):
|
||||||
|
continue
|
||||||
|
retired_path = immutable_path[len(marker):]
|
||||||
|
else:
|
||||||
|
retired_path = immutable_path
|
||||||
|
collision = html_dir / retired_path
|
||||||
|
if collision.exists() and any(path.is_file() for path in collision.rglob("*")):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"current artifact collides with retired immutable partition: {retired_path}"
|
||||||
|
)
|
||||||
|
filters.extend(("--exclude", f"{retired_path}/*"))
|
||||||
|
return filters
|
||||||
|
|
||||||
|
|
||||||
|
def s3_sync(artifact, route, site_dir, credential_env_names=None, previous_contract=None):
|
||||||
|
endpoint = artifact["s3_endpoint"]
|
||||||
|
html_dir = site_dir / artifact["build_dir"]
|
||||||
|
aws_env = publication_aws_env(artifact, credential_env_names)
|
||||||
bucket = artifact["bucket"]
|
bucket = artifact["bucket"]
|
||||||
object_prefix = route["path"].strip("/")
|
object_prefix = route["path"].strip("/")
|
||||||
bucket_destination = f"s3://{bucket}/"
|
destination = f"s3://{bucket}/{object_prefix + '/' if object_prefix else ''}"
|
||||||
destination = f"{bucket_destination}{object_prefix + '/' if object_prefix else ''}"
|
|
||||||
default_cache = next(rule["cache_control"] for rule in artifact["cache_rules"] if not rule["path"])
|
default_cache = next(rule["cache_control"] for rule in artifact["cache_rules"] if not rule["path"])
|
||||||
|
immutable_paths = [rule["path"] for rule in artifact["cache_rules"] if _is_immutable(rule)]
|
||||||
# `excludes` are patterns (site.yaml `excludes:` list) that should never
|
# `excludes` are patterns (site.yaml `excludes:` list) that should never
|
||||||
# be uploaded *and* should never be deleted from the bucket — escape hatch
|
# be uploaded *and* should never be deleted from the bucket — escape hatch
|
||||||
# for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli).
|
# for assets managed out-of-band (e.g. large PDFs uploaded via aws-cli).
|
||||||
@@ -75,41 +330,30 @@ def s3_sync(artifact, route, site_dir, credential_env_names=None):
|
|||||||
if artifact["excludes"]:
|
if artifact["excludes"]:
|
||||||
print(f"Excluding patterns: {artifact['excludes']}")
|
print(f"Excluding patterns: {artifact['excludes']}")
|
||||||
print(f"Syncing artifact {artifact['name']} → {destination} via {endpoint}")
|
print(f"Syncing artifact {artifact['name']} → {destination} via {endpoint}")
|
||||||
# `sync --delete` handles new/changed/orphaned files. Partitioned
|
# Upload with the final cache policy before cleanup. Sync and deletion are
|
||||||
# `cp --recursive` calls then re-upload each file once to refresh metadata
|
# scoped to the same current route prefix and cache partition. A route move
|
||||||
# (cache-control, content-type) on objects sync skipped as unchanged.
|
# leaves its old bucket partition intact but unreachable after the old
|
||||||
|
# Ingress disappears, while stale mutable keys on the serving prefix are
|
||||||
|
# deleted. Immutable subtrees are structurally excluded. `cp --recursive`
|
||||||
|
# refreshes metadata atomically per object before `sync --delete` removes
|
||||||
|
# stale keys without ever exposing new bytes under a provisional policy.
|
||||||
# A no-op deploy therefore transfers the artifact bytes once.
|
# A no-op deploy therefore transfers the artifact bytes once.
|
||||||
# AWS CLI guesses Content-Type from file extension on local→S3 uploads,
|
# AWS CLI guesses Content-Type from file extension on local→S3 uploads,
|
||||||
# so a fresh upload always carries the right MIME type.
|
# so a fresh upload always carries the right MIME type.
|
||||||
stage = None
|
|
||||||
sync_source = html_dir
|
|
||||||
sync_excludes = exclude_args
|
|
||||||
if object_prefix:
|
|
||||||
stage = tempfile.TemporaryDirectory()
|
|
||||||
sync_source = Path(stage.name)
|
|
||||||
staged_artifact = sync_source / object_prefix
|
|
||||||
staged_artifact.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
staged_artifact.symlink_to(html_dir.resolve(), target_is_directory=True)
|
|
||||||
sync_excludes = [arg for pattern in artifact["excludes"]
|
|
||||||
for arg in ("--exclude", f"{object_prefix}/{pattern}")]
|
|
||||||
try:
|
|
||||||
# Sync the complete bucket authority so moving a route prefix also
|
|
||||||
# deletes objects under its old prefix instead of leaving them public.
|
|
||||||
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{sync_source}/", bucket_destination,
|
|
||||||
"--delete", "--only-show-errors", "--cache-control", default_cache,
|
|
||||||
*sync_excludes], env=aws_env)
|
|
||||||
finally:
|
|
||||||
if stage:
|
|
||||||
stage.cleanup()
|
|
||||||
print("Re-stamping metadata on all objects...")
|
|
||||||
specific_paths = [rule["path"] for rule in artifact["cache_rules"] if rule["path"]]
|
specific_paths = [rule["path"] for rule in artifact["cache_rules"] if rule["path"]]
|
||||||
default_filters = [arg for path in specific_paths for arg in ("--exclude", f"{path}/*")]
|
default_filters = [arg for path in specific_paths for arg in ("--exclude", f"{path}/*")]
|
||||||
|
retired_filters = retired_immutable_filters(artifact, route, html_dir, previous_contract)
|
||||||
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
||||||
"--recursive", "--only-show-errors", "--cache-control", default_cache,
|
"--recursive", "--only-show-errors", "--cache-control", default_cache,
|
||||||
*default_filters, *exclude_args], env=aws_env)
|
*default_filters, *retired_filters, *exclude_args], env=aws_env)
|
||||||
|
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
|
||||||
|
"--delete", "--only-show-errors", "--cache-control", default_cache,
|
||||||
|
*default_filters, *retired_filters, *exclude_args], env=aws_env)
|
||||||
for rule in artifact["cache_rules"]:
|
for rule in artifact["cache_rules"]:
|
||||||
if not rule["path"]:
|
if not rule["path"]:
|
||||||
continue
|
continue
|
||||||
|
if _is_immutable(rule):
|
||||||
|
continue
|
||||||
include = f"{rule['path'].rstrip('/')}/*"
|
include = f"{rule['path'].rstrip('/')}/*"
|
||||||
child_filters = [arg for path in specific_paths
|
child_filters = [arg for path in specific_paths
|
||||||
if path.startswith(f"{rule['path'].rstrip('/')}/")
|
if path.startswith(f"{rule['path'].rstrip('/')}/")
|
||||||
@@ -119,6 +363,9 @@ def s3_sync(artifact, route, site_dir, credential_env_names=None):
|
|||||||
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
run(["aws", "--endpoint-url", endpoint, "s3", "cp", f"{html_dir}/", destination,
|
||||||
"--recursive", "--only-show-errors", "--cache-control", rule["cache_control"],
|
"--recursive", "--only-show-errors", "--cache-control", rule["cache_control"],
|
||||||
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
|
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
|
||||||
|
run(["aws", "--endpoint-url", endpoint, "s3", "sync", f"{html_dir}/", destination,
|
||||||
|
"--delete", "--only-show-errors", "--cache-control", rule["cache_control"],
|
||||||
|
"--exclude", "*", "--include", include, *child_filters, *exclude_args], env=aws_env)
|
||||||
|
|
||||||
|
|
||||||
def garage_admin(method, path, token, body=None):
|
def garage_admin(method, path, token, body=None):
|
||||||
@@ -167,16 +414,32 @@ def ensure_bucket_aliases(site_name, aliases, admin_token):
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
|
|
||||||
def render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg):
|
def render_site_manifests(
|
||||||
|
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts=None,
|
||||||
|
):
|
||||||
"""Always re-render manifests from current site.yaml. Templates own
|
"""Always re-render manifests from current site.yaml. Templates own
|
||||||
domain + aliases, so changes propagate without manual edits."""
|
domain + aliases, so changes propagate without manual edits."""
|
||||||
manifests_dir.mkdir(parents=True, exist_ok=True)
|
manifests_dir.mkdir(parents=True, exist_ok=True)
|
||||||
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
routes = []
|
routes = []
|
||||||
|
previous_contracts = previous_contracts or {}
|
||||||
|
next_contracts = {bucket: dict(contract)
|
||||||
|
for bucket, contract in previous_contracts.items()}
|
||||||
for route in cfg["routes"]:
|
for route in cfg["routes"]:
|
||||||
artifact = artifact_by_name[route["artifact"]]
|
artifact = artifact_by_name[route["artifact"]]
|
||||||
resource_name = k8s_name(site_name) if cfg["compatibility"] else f"{k8s_name(site_name)}-{route['name']}"
|
resource_name = k8s_name(site_name) if cfg["compatibility"] else f"{k8s_name(site_name)}-{route['name']}"
|
||||||
routes.append({**route, "resource_name": resource_name, "artifact_config": artifact})
|
previous = previous_contracts.get(artifact["bucket"])
|
||||||
|
immutable_paths = retained_immutable_paths(artifact, route, previous)
|
||||||
|
next_contracts[artifact["bucket"]] = {
|
||||||
|
"path": route["path"],
|
||||||
|
"access": "public" if route["access"] == "legacy" else route["access"],
|
||||||
|
"artifact": route["artifact"],
|
||||||
|
"immutable_paths": immutable_paths,
|
||||||
|
}
|
||||||
|
routes.append({
|
||||||
|
**route, "resource_name": resource_name, "artifact_config": artifact,
|
||||||
|
"immutable_paths_json": json.dumps(immutable_paths, separators=(",", ":")),
|
||||||
|
})
|
||||||
template_vars = {
|
template_vars = {
|
||||||
"site": site_name,
|
"site": site_name,
|
||||||
"site_k8s": k8s_name(site_name),
|
"site_k8s": k8s_name(site_name),
|
||||||
@@ -187,6 +450,126 @@ def render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg):
|
|||||||
"routes": routes,
|
"routes": routes,
|
||||||
}
|
}
|
||||||
render_templates(action_dir, template_vars, app_dir, manifests_dir)
|
render_templates(action_dir, template_vars, app_dir, manifests_dir)
|
||||||
|
if not cfg["compatibility"] or previous_contracts:
|
||||||
|
(app_dir / "site-publish-history.yaml").write_text(yaml.safe_dump(
|
||||||
|
{"version": 1, "buckets": next_contracts}, sort_keys=True,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_route_contract(bucket, contract, path):
|
||||||
|
expected = {"path", "access", "artifact", "immutable_paths"}
|
||||||
|
if (not isinstance(bucket, str) or not bucket or not isinstance(contract, dict)
|
||||||
|
or set(contract) != expected
|
||||||
|
or contract.get("access") not in {"public", "protected"}
|
||||||
|
or not isinstance(contract.get("path"), str)
|
||||||
|
or not contract["path"].startswith("/")
|
||||||
|
or not isinstance(contract.get("artifact"), str) or not contract["artifact"]
|
||||||
|
or not isinstance(contract.get("immutable_paths"), list)
|
||||||
|
or any(not isinstance(item, str) or not item or item.startswith("/")
|
||||||
|
or any(part in {"", ".", ".."} for part in item.split("/"))
|
||||||
|
for item in contract["immutable_paths"])):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}")
|
||||||
|
return {
|
||||||
|
"path": contract["path"], "access": contract["access"],
|
||||||
|
"artifact": contract["artifact"],
|
||||||
|
"immutable_paths": sorted(set(contract["immutable_paths"])),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def previous_route_contracts(app_dir):
|
||||||
|
"""Read bucket-keyed route history from generated Ingresses."""
|
||||||
|
history_path = app_dir / "site-publish-history.yaml"
|
||||||
|
if history_path.exists():
|
||||||
|
document = yaml.safe_load(history_path.read_text())
|
||||||
|
if (not isinstance(document, dict) or set(document) != {"version", "buckets"}
|
||||||
|
or document["version"] != 1 or not isinstance(document["buckets"], dict)):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {history_path}")
|
||||||
|
return {
|
||||||
|
bucket: _validate_route_contract(bucket, contract, history_path)
|
||||||
|
for bucket, contract in document["buckets"].items()
|
||||||
|
}
|
||||||
|
contracts = {}
|
||||||
|
manifests = app_dir / "manifests"
|
||||||
|
if not manifests.exists():
|
||||||
|
return contracts
|
||||||
|
for path in sorted(manifests.glob("ingress*.yaml")):
|
||||||
|
document = yaml.safe_load(path.read_text()) or {}
|
||||||
|
annotations = document.get("metadata", {}).get("annotations", {})
|
||||||
|
artifact = annotations.get("site-publish.fritzlab.net/artifact")
|
||||||
|
access = annotations.get("site-publish.fritzlab.net/access")
|
||||||
|
bucket = annotations.get("site-publish.fritzlab.net/bucket")
|
||||||
|
immutable_paths_json = annotations.get("site-publish.fritzlab.net/immutable-paths")
|
||||||
|
route_path = annotations.get("site-publish.fritzlab.net/route-path")
|
||||||
|
values = (artifact, access, bucket, immutable_paths_json, route_path)
|
||||||
|
if all(value is None for value in values):
|
||||||
|
continue
|
||||||
|
if (not all(isinstance(value, str) for value in values)
|
||||||
|
or access not in {"public", "protected"} or not route_path.startswith("/")):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}")
|
||||||
|
try:
|
||||||
|
immutable_paths = json.loads(immutable_paths_json)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}") from exc
|
||||||
|
if not isinstance(immutable_paths, list) or any(not isinstance(item, str) for item in immutable_paths):
|
||||||
|
raise RuntimeError(f"invalid site-publish route history in {path}")
|
||||||
|
if bucket in contracts:
|
||||||
|
raise RuntimeError(f"duplicate site-publish route history for bucket {bucket}")
|
||||||
|
contracts[bucket] = _validate_route_contract(bucket, {
|
||||||
|
"path": route_path, "access": access, "artifact": artifact,
|
||||||
|
"immutable_paths": immutable_paths,
|
||||||
|
}, path)
|
||||||
|
return contracts
|
||||||
|
|
||||||
|
|
||||||
|
def validate_route_migrations(cfg, previous_contracts):
|
||||||
|
artifacts = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
artifact = artifacts[route["artifact"]]
|
||||||
|
previous = previous_contracts.get(artifact["bucket"])
|
||||||
|
if (previous and previous["access"] == "protected"
|
||||||
|
and route["access"] in {"public", "legacy"}
|
||||||
|
):
|
||||||
|
raise RuntimeError(
|
||||||
|
f"artifact {route['artifact']} cannot become public while reusing protected "
|
||||||
|
f"bucket {artifact['bucket']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_scoped_history(cfg, previous_contracts):
|
||||||
|
"""A scoped run may not record a route contract it did not publish.
|
||||||
|
|
||||||
|
render_site_manifests advances the stored contract for every route in
|
||||||
|
site.yaml, and `access` is overwritten rather than unioned the way
|
||||||
|
immutable_paths is. Without this, a catalogue-only publish could write a
|
||||||
|
protected access for the distributions bucket that nothing published, and
|
||||||
|
validate_route_migrations would then refuse to put that bucket back —
|
||||||
|
unpublished intent turned into an irreversible fact.
|
||||||
|
"""
|
||||||
|
chosen = set(cfg["selected"])
|
||||||
|
if len(chosen) == len(cfg["artifacts"]):
|
||||||
|
return
|
||||||
|
artifact_by_name = {artifact["name"]: artifact for artifact in cfg["artifacts"]}
|
||||||
|
for route in cfg["routes"]:
|
||||||
|
if route["artifact"] in chosen:
|
||||||
|
continue
|
||||||
|
artifact = artifact_by_name[route["artifact"]]
|
||||||
|
previous = previous_contracts.get(artifact["bucket"])
|
||||||
|
if previous is None:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"a scoped run cannot introduce artifact {route['artifact']}; "
|
||||||
|
f"publish it in the same run"
|
||||||
|
)
|
||||||
|
current = {
|
||||||
|
"path": route["path"],
|
||||||
|
"access": "public" if route["access"] == "legacy" else route["access"],
|
||||||
|
"artifact": route["artifact"],
|
||||||
|
}
|
||||||
|
recorded = {key: previous[key] for key in current}
|
||||||
|
if recorded != current:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"a scoped run may not change unselected artifact {route['artifact']}'s route "
|
||||||
|
f"contract ({recorded} -> {current}); publish it in the same run"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def deploy_static(site_name, site_dir, action_dir, token, cfg):
|
def deploy_static(site_name, site_dir, action_dir, token, cfg):
|
||||||
@@ -194,21 +577,43 @@ def deploy_static(site_name, site_dir, action_dir, token, cfg):
|
|||||||
credential_env_names = {
|
credential_env_names = {
|
||||||
name for artifact in cfg["artifacts"] for name in artifact["credentials"].values()
|
name for artifact in cfg["artifacts"] for name in artifact["credentials"].values()
|
||||||
}
|
}
|
||||||
|
# Publication is scoped to the selected artifacts; the rendered route
|
||||||
|
# contract is not. Manifests and immutable-path history stay whole, so a
|
||||||
|
# partial publish can never retire another artifact's route or bucket.
|
||||||
|
publishing = selected_routes(cfg)
|
||||||
validate_publication_environment(cfg)
|
validate_publication_environment(cfg)
|
||||||
for artifact in cfg["artifacts"]:
|
for artifact in selected_artifacts(cfg):
|
||||||
validate_artifact_output(site_dir, artifact)
|
validate_artifact_output(site_dir, artifact)
|
||||||
for route in cfg["routes"]:
|
|
||||||
s3_sync(artifact_by_name[route["artifact"]], route, site_dir, credential_env_names)
|
|
||||||
if cfg["compatibility"]:
|
|
||||||
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
|
|
||||||
|
|
||||||
apps_dir = clone_apps(token)
|
apps_dir = clone_apps(token)
|
||||||
app_dir = apps_dir / "sjc001" / "websites" / site_name
|
app_dir = apps_dir / "sjc001" / "websites" / site_name
|
||||||
manifests_dir = app_dir / "manifests"
|
manifests_dir = app_dir / "manifests"
|
||||||
|
previous_contracts = previous_route_contracts(app_dir)
|
||||||
|
validate_route_migrations(cfg, previous_contracts)
|
||||||
|
validate_scoped_history(cfg, previous_contracts)
|
||||||
|
# Complete immutable work across the whole publication before any route's
|
||||||
|
# mutable pointers can change. Partial immutable success is safe; mixing a
|
||||||
|
# new route with an old route after a later immutable failure is not.
|
||||||
|
for route in publishing:
|
||||||
|
publish_route_immutables(
|
||||||
|
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
|
||||||
|
)
|
||||||
|
# Reconcile every browser-read policy before publishing mutable content.
|
||||||
|
# A CORS failure therefore cannot leave a new channel pointing at a release
|
||||||
|
# whose cross-origin assets browsers cannot consume.
|
||||||
|
reconcile_artifact_cors(selected_artifacts(cfg), credential_env_names)
|
||||||
|
for route in publishing:
|
||||||
|
s3_sync(
|
||||||
|
artifact_by_name[route["artifact"]], route, site_dir, credential_env_names,
|
||||||
|
previous_contracts.get(artifact_by_name[route["artifact"]]["bucket"]),
|
||||||
|
)
|
||||||
|
if cfg["compatibility"]:
|
||||||
|
ensure_bucket_aliases(site_name, cfg["aliases"], os.environ.get("GARAGE_ADMIN_TOKEN"))
|
||||||
|
|
||||||
render_site_manifests(site_name, action_dir, app_dir, manifests_dir, cfg)
|
render_site_manifests(
|
||||||
|
site_name, action_dir, app_dir, manifests_dir, cfg, previous_contracts,
|
||||||
|
)
|
||||||
|
|
||||||
commit_and_push(apps_dir, f"Deploy {site_name}")
|
commit_and_push(apps_dir, f"Deploy {site_name}", token)
|
||||||
|
|
||||||
|
|
||||||
def decommission(site_name, token, buckets=None):
|
def decommission(site_name, token, buckets=None):
|
||||||
@@ -219,7 +624,7 @@ def decommission(site_name, token, buckets=None):
|
|||||||
print(f"No manifests for {site_name} — nothing to remove")
|
print(f"No manifests for {site_name} — nothing to remove")
|
||||||
return
|
return
|
||||||
shutil.rmtree(site_path)
|
shutil.rmtree(site_path)
|
||||||
commit_and_push(apps_dir, f"Decommission {site_name}")
|
commit_and_push(apps_dir, f"Decommission {site_name}", token)
|
||||||
for bucket in buckets or [site_name]:
|
for bucket in buckets or [site_name]:
|
||||||
print(f"Bucket {bucket} and its objects are NOT purged automatically.")
|
print(f"Bucket {bucket} and its objects are NOT purged automatically.")
|
||||||
print(f" garage bucket delete {bucket} --yes")
|
print(f" garage bucket delete {bucket} --yes")
|
||||||
@@ -235,8 +640,11 @@ def cmd_deploy():
|
|||||||
cfg = parse_site_yaml(site_dir)
|
cfg = parse_site_yaml(site_dir)
|
||||||
|
|
||||||
if not cfg["enabled"]:
|
if not cfg["enabled"]:
|
||||||
|
if len(cfg["selected"]) != len(cfg["artifacts"]):
|
||||||
|
die("a disabled site decommissions whole; drop the artifacts selection")
|
||||||
print("Site disabled — running decommission...")
|
print("Site disabled — running decommission...")
|
||||||
decommission(site_name, token, [artifact["bucket"] for artifact in cfg["artifacts"]])
|
decommission(site_name, token, [artifact["bucket"] for artifact in cfg["artifacts"]])
|
||||||
return
|
return
|
||||||
|
|
||||||
|
validate_artifact_inputs(site_dir, cfg)
|
||||||
deploy_static(site_name, site_dir, action_dir, token, cfg)
|
deploy_static(site_name, site_dir, action_dir, token, cfg)
|
||||||
|
|||||||
+165
-13
@@ -1,5 +1,6 @@
|
|||||||
"""Shared utilities for the site-publish action."""
|
"""Shared utilities for the site-publish action."""
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
@@ -26,6 +27,7 @@ EXCLUDE_FILES = {
|
|||||||
VALID_TYPES = {"static", "hugo", "mkdocs"}
|
VALID_TYPES = {"static", "hugo", "mkdocs"}
|
||||||
NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")
|
NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")
|
||||||
ENV_RE = re.compile(r"^[A-Z_][A-Z0-9_]*$")
|
ENV_RE = re.compile(r"^[A-Z_][A-Z0-9_]*$")
|
||||||
|
ACCESS_KEY_ENV_RE = re.compile(r"^([A-Z][A-Z0-9_]*)_S3_ACCESS_KEY$")
|
||||||
BUCKET_RE = re.compile(r"^[a-z0-9](?:[a-z0-9.-]{1,61}[a-z0-9])?$")
|
BUCKET_RE = re.compile(r"^[a-z0-9](?:[a-z0-9.-]{1,61}[a-z0-9])?$")
|
||||||
MIDDLEWARE_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9])?$")
|
MIDDLEWARE_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9])?$")
|
||||||
|
|
||||||
@@ -103,6 +105,55 @@ def _strings(value, label):
|
|||||||
return values
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def _cors_origins(value, label):
|
||||||
|
origins = _list(value, label)
|
||||||
|
if any(not isinstance(item, str) or not item for item in origins):
|
||||||
|
raise ConfigError(f"{label} must be a list of non-empty strings")
|
||||||
|
canonical = []
|
||||||
|
for origin in origins:
|
||||||
|
if origin == "*":
|
||||||
|
canonical.append(origin)
|
||||||
|
continue
|
||||||
|
parsed = urlparse(origin)
|
||||||
|
try:
|
||||||
|
port = parsed.port
|
||||||
|
except ValueError:
|
||||||
|
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins") from None
|
||||||
|
if parsed.scheme != "https" or not parsed.hostname or parsed.path or parsed.params or (
|
||||||
|
parsed.query or parsed.fragment or parsed.username or parsed.password
|
||||||
|
):
|
||||||
|
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
|
||||||
|
try:
|
||||||
|
address = ipaddress.ip_address(parsed.hostname)
|
||||||
|
except ValueError:
|
||||||
|
try:
|
||||||
|
hostname = parsed.hostname.encode("idna").decode("ascii")
|
||||||
|
except UnicodeError:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{label} must contain '*' or canonical HTTPS origins"
|
||||||
|
) from None
|
||||||
|
try:
|
||||||
|
_hostname(hostname, f"{label} hostname")
|
||||||
|
except ConfigError:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{label} must contain '*' or canonical HTTPS origins"
|
||||||
|
) from None
|
||||||
|
else:
|
||||||
|
if getattr(address, "scope_id", None) is not None:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{label} must contain '*' or canonical HTTPS origins"
|
||||||
|
)
|
||||||
|
hostname = f"[{address.compressed}]" if address.version == 6 else address.compressed
|
||||||
|
canonical.append(f"https://{hostname}{f':{port}' if port not in (None, 443) else ''}")
|
||||||
|
if len(canonical) != len(set(canonical)):
|
||||||
|
raise ConfigError(f"{label} must not contain duplicate canonical origins")
|
||||||
|
if "*" in canonical and len(canonical) != 1:
|
||||||
|
raise ConfigError(f"{label} wildcard must be the only origin")
|
||||||
|
if origins != canonical:
|
||||||
|
raise ConfigError(f"{label} must contain '*' or canonical HTTPS origins")
|
||||||
|
return origins
|
||||||
|
|
||||||
|
|
||||||
def _hostname(value, label):
|
def _hostname(value, label):
|
||||||
if not isinstance(value, str) or len(value) > 253 or value.endswith("."):
|
if not isinstance(value, str) or len(value) > 253 or value.endswith("."):
|
||||||
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
|
raise ConfigError(f"{label} must be a lowercase DNS hostname without a trailing dot")
|
||||||
@@ -214,6 +265,7 @@ def _legacy_config(raw, site_name):
|
|||||||
"website_authority": "garage-s3.storage.svc.k8s.sjc001.fritzlab.net",
|
"website_authority": "garage-s3.storage.svc.k8s.sjc001.fritzlab.net",
|
||||||
"credentials": {"access_key_env": "AWS_ACCESS_KEY_ID", "secret_key_env": "AWS_SECRET_ACCESS_KEY"},
|
"credentials": {"access_key_env": "AWS_ACCESS_KEY_ID", "secret_key_env": "AWS_SECRET_ACCESS_KEY"},
|
||||||
"cache_rules": [{"path": "", "cache_control": DEFAULT_CACHE_CONTROL}],
|
"cache_rules": [{"path": "", "cache_control": DEFAULT_CACHE_CONTROL}],
|
||||||
|
"cors_origins": None,
|
||||||
}
|
}
|
||||||
return {
|
return {
|
||||||
"version": 1,
|
"version": 1,
|
||||||
@@ -222,6 +274,7 @@ def _legacy_config(raw, site_name):
|
|||||||
"aliases": _aliases(raw.get("aliases"), raw["domain"]),
|
"aliases": _aliases(raw.get("aliases"), raw["domain"]),
|
||||||
"enabled": raw.get("enabled", True),
|
"enabled": raw.get("enabled", True),
|
||||||
"artifacts": [artifact],
|
"artifacts": [artifact],
|
||||||
|
"selected": ["site"],
|
||||||
"routes": [{
|
"routes": [{
|
||||||
"name": "site", "path": "/", "artifact": "site", "access": "legacy",
|
"name": "site", "path": "/", "artifact": "site", "access": "legacy",
|
||||||
"access_middleware": None,
|
"access_middleware": None,
|
||||||
@@ -233,12 +286,16 @@ def _legacy_config(raw, site_name):
|
|||||||
def _artifact(item, index):
|
def _artifact(item, index):
|
||||||
label = f"artifacts[{index}]"
|
label = f"artifacts[{index}]"
|
||||||
item = _mapping(item, label)
|
item = _mapping(item, label)
|
||||||
_known_keys(item, {"name", "type", "content_dir", "tidy", "excludes", "publish", "cache"}, label)
|
_known_keys(
|
||||||
|
item,
|
||||||
|
{"name", "type", "content_dir", "tidy", "excludes", "publish", "cache", "cors_origins"},
|
||||||
|
label,
|
||||||
|
)
|
||||||
name = item.get("name")
|
name = item.get("name")
|
||||||
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
|
if not isinstance(name, str) or not NAME_RE.fullmatch(name):
|
||||||
raise ConfigError(f"{label}.name must be a DNS label")
|
raise ConfigError(f"{label}.name must be a DNS label")
|
||||||
publish = _mapping(item.get("publish"), f"{label}.publish")
|
publish = _mapping(item.get("publish"), f"{label}.publish")
|
||||||
_known_keys(publish, {"bucket", "endpoint", "website_authority", "credentials"}, f"{label}.publish")
|
_known_keys(publish, {"bucket", "credentials"}, f"{label}.publish")
|
||||||
bucket = publish.get("bucket")
|
bucket = publish.get("bucket")
|
||||||
if not isinstance(bucket, str) or not BUCKET_RE.fullmatch(bucket):
|
if not isinstance(bucket, str) or not BUCKET_RE.fullmatch(bucket):
|
||||||
raise ConfigError(f"{label}.publish.bucket is not a valid bucket name")
|
raise ConfigError(f"{label}.publish.bucket is not a valid bucket name")
|
||||||
@@ -250,6 +307,15 @@ def _artifact(item, index):
|
|||||||
if not isinstance(value, str) or not ENV_RE.fullmatch(value):
|
if not isinstance(value, str) or not ENV_RE.fullmatch(value):
|
||||||
raise ConfigError(f"{label}.publish.credentials.{key} must name an environment variable")
|
raise ConfigError(f"{label}.publish.credentials.{key} must name an environment variable")
|
||||||
normalized_credentials[key] = value
|
normalized_credentials[key] = value
|
||||||
|
access_match = ACCESS_KEY_ENV_RE.fullmatch(normalized_credentials["access_key_env"])
|
||||||
|
expected_secret = (
|
||||||
|
f"{access_match.group(1)}_S3_SECRET_KEY" if access_match else None
|
||||||
|
)
|
||||||
|
if normalized_credentials["secret_key_env"] != expected_secret:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{label}.publish.credentials must be a matched "
|
||||||
|
"<NAME>_S3_ACCESS_KEY and <NAME>_S3_SECRET_KEY pair"
|
||||||
|
)
|
||||||
cache = _mapping(item.get("cache"), f"{label}.cache")
|
cache = _mapping(item.get("cache"), f"{label}.cache")
|
||||||
_known_keys(cache, {"rules"}, f"{label}.cache")
|
_known_keys(cache, {"rules"}, f"{label}.cache")
|
||||||
rules = _list(cache.get("rules"), f"{label}.cache.rules")
|
rules = _list(cache.get("rules"), f"{label}.cache.rules")
|
||||||
@@ -270,10 +336,15 @@ def _artifact(item, index):
|
|||||||
if "" not in paths:
|
if "" not in paths:
|
||||||
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
|
raise ConfigError(f"{label}.cache.rules must declare a '/' default")
|
||||||
cache_rules.sort(key=lambda rule: (len(PurePosixPath(rule["path"]).parts), rule["path"]))
|
cache_rules.sort(key=lambda rule: (len(PurePosixPath(rule["path"]).parts), rule["path"]))
|
||||||
authority = _hostname(
|
immutable_paths = [rule["path"] for rule in cache_rules
|
||||||
publish.get("website_authority", f"{bucket}.{DEFAULT_WEBSITE_SUFFIX}"),
|
if "immutable" in {part.strip().lower().split("=", 1)[0]
|
||||||
f"{label}.publish.website_authority",
|
for part in rule["cache_control"].split(",")}]
|
||||||
)
|
if "" in immutable_paths:
|
||||||
|
raise ConfigError(f"{label}.cache.rules immutable paths must be narrower than '/'")
|
||||||
|
for path in immutable_paths:
|
||||||
|
if any(other.startswith(f"{path}/") for other in paths):
|
||||||
|
raise ConfigError(f"{label}.cache.rules cannot nest another policy under immutable /{path}")
|
||||||
|
authority = f"{bucket}.{DEFAULT_WEBSITE_SUFFIX}"
|
||||||
if not isinstance(item.get("tidy", True), bool):
|
if not isinstance(item.get("tidy", True), bool):
|
||||||
raise ConfigError(f"{label}.tidy must be a boolean")
|
raise ConfigError(f"{label}.tidy must be a boolean")
|
||||||
return {
|
return {
|
||||||
@@ -284,10 +355,14 @@ def _artifact(item, index):
|
|||||||
"excludes": _strings(item.get("excludes") or [], f"{label}.excludes"),
|
"excludes": _strings(item.get("excludes") or [], f"{label}.excludes"),
|
||||||
"build_dir": f".site-publish/{name}/html",
|
"build_dir": f".site-publish/{name}/html",
|
||||||
"bucket": bucket,
|
"bucket": bucket,
|
||||||
"s3_endpoint": _endpoint(publish.get("endpoint", DEFAULT_S3_ENDPOINT), f"{label}.publish.endpoint"),
|
"s3_endpoint": DEFAULT_S3_ENDPOINT,
|
||||||
"website_authority": authority,
|
"website_authority": authority,
|
||||||
"credentials": normalized_credentials,
|
"credentials": normalized_credentials,
|
||||||
"cache_rules": cache_rules,
|
"cache_rules": cache_rules,
|
||||||
|
"cors_origins": (
|
||||||
|
_cors_origins(item["cors_origins"], f"{label}.cors_origins")
|
||||||
|
if "cors_origins" in item else []
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -374,11 +449,42 @@ def _validate_multi(cfg):
|
|||||||
raise ConfigError(f"protected route {route['name']} cannot use shared-cache max-age")
|
raise ConfigError(f"protected route {route['name']} cannot use shared-cache max-age")
|
||||||
if route["access"] == "public" and "private" in directives:
|
if route["access"] == "public" and "private" in directives:
|
||||||
raise ConfigError(f"public route {route['name']} cannot use private cache policy")
|
raise ConfigError(f"public route {route['name']} cannot use private cache policy")
|
||||||
|
if route["access"] == "protected" and "*" in artifact["cors_origins"]:
|
||||||
|
raise ConfigError(
|
||||||
|
f"protected route {route['name']} cannot allow wildcard CORS"
|
||||||
|
)
|
||||||
root = next(route for route in routes if route["path"] == "/")
|
root = next(route for route in routes if route["path"] == "/")
|
||||||
if any(route["access"] == "protected" for route in routes) and root["access"] == "public":
|
if any(route["access"] == "protected" for route in routes) and root["access"] == "public":
|
||||||
raise ConfigError("a public '/' catch-all would expose unmatched protected content")
|
raise ConfigError("a public '/' catch-all would expose unmatched protected content")
|
||||||
|
|
||||||
|
|
||||||
|
def _artifact_selection(cfg, requested):
|
||||||
|
"""Resolve the subset of declared artifacts this run publishes."""
|
||||||
|
declared = [artifact["name"] for artifact in cfg["artifacts"]]
|
||||||
|
if not requested or not requested.strip():
|
||||||
|
return declared
|
||||||
|
names = [part for part in re.split(r"[,\s]+", requested.strip()) if part]
|
||||||
|
unknown = sorted({name for name in names if name not in declared})
|
||||||
|
if unknown:
|
||||||
|
raise ConfigError(
|
||||||
|
"artifacts selects undeclared artifact(s) " + ", ".join(unknown)
|
||||||
|
+ "; site.yaml declares " + ", ".join(declared)
|
||||||
|
)
|
||||||
|
return [name for name in declared if name in set(names)]
|
||||||
|
|
||||||
|
|
||||||
|
def selected_artifacts(cfg):
|
||||||
|
"""Artifacts this run builds and publishes, in declaration order."""
|
||||||
|
chosen = set(cfg["selected"])
|
||||||
|
return [artifact for artifact in cfg["artifacts"] if artifact["name"] in chosen]
|
||||||
|
|
||||||
|
|
||||||
|
def selected_routes(cfg):
|
||||||
|
"""Routes whose artifact this run publishes, in route order."""
|
||||||
|
chosen = set(cfg["selected"])
|
||||||
|
return [route for route in cfg["routes"] if route["artifact"] in chosen]
|
||||||
|
|
||||||
|
|
||||||
def normalize_site_config(raw, site_name):
|
def normalize_site_config(raw, site_name):
|
||||||
raw = _mapping(raw, "site.yaml")
|
raw = _mapping(raw, "site.yaml")
|
||||||
domain = _hostname(raw.get("domain"), "domain")
|
domain = _hostname(raw.get("domain"), "domain")
|
||||||
@@ -404,12 +510,49 @@ def normalize_site_config(raw, site_name):
|
|||||||
"routes": sorted(routes, key=lambda item: (-len(item["path"]), item["path"], item["name"])),
|
"routes": sorted(routes, key=lambda item: (-len(item["path"]), item["path"], item["name"])),
|
||||||
}
|
}
|
||||||
_validate_multi(cfg)
|
_validate_multi(cfg)
|
||||||
|
cfg["selected"] = [artifact["name"] for artifact in cfg["artifacts"]]
|
||||||
for route in cfg["routes"]:
|
for route in cfg["routes"]:
|
||||||
if len(f"{k8s_name(site_name)}-{route['name']}") > 63:
|
if len(f"{k8s_name(site_name)}-{route['name']}") > 63:
|
||||||
raise ConfigError(f"route {route['name']} makes the generated Service name exceed 63 characters")
|
raise ConfigError(f"route {route['name']} makes the generated Service name exceed 63 characters")
|
||||||
return cfg
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def validate_artifact_inputs(site_dir, cfg):
|
||||||
|
"""Reject source containment before a public or protected build starts."""
|
||||||
|
if cfg["compatibility"]:
|
||||||
|
return
|
||||||
|
root = Path(site_dir).resolve()
|
||||||
|
sources = []
|
||||||
|
for artifact in cfg["artifacts"]:
|
||||||
|
declared = root
|
||||||
|
for component in Path(artifact["content_dir"]).parts:
|
||||||
|
declared /= component
|
||||||
|
if declared.is_symlink():
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} content_dir contains symlink component: "
|
||||||
|
f"{declared.relative_to(root)}"
|
||||||
|
)
|
||||||
|
source = declared.resolve()
|
||||||
|
if source != root and root not in source.parents:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} content_dir resolves outside the repository"
|
||||||
|
)
|
||||||
|
if source.exists():
|
||||||
|
symlink = next((path for path in source.rglob("*") if path.is_symlink()), None)
|
||||||
|
if symlink is not None:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact {artifact['name']} build input contains symlink: "
|
||||||
|
f"{symlink.relative_to(root)}"
|
||||||
|
)
|
||||||
|
sources.append((artifact["name"], source))
|
||||||
|
for index, (name, source) in enumerate(sources):
|
||||||
|
for other_name, other_source in sources[index + 1:]:
|
||||||
|
if source == other_source or source in other_source.parents or other_source in source.parents:
|
||||||
|
raise ConfigError(
|
||||||
|
f"artifact build inputs overlap after resolution: {name} and {other_name}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def parse_site_yaml(site_dir, site_name=None):
|
def parse_site_yaml(site_dir, site_name=None):
|
||||||
path = Path(site_dir) / "site.yaml"
|
path = Path(site_dir) / "site.yaml"
|
||||||
if not path.exists():
|
if not path.exists():
|
||||||
@@ -420,12 +563,14 @@ def parse_site_yaml(site_dir, site_name=None):
|
|||||||
try:
|
try:
|
||||||
with open(path) as stream:
|
with open(path) as stream:
|
||||||
cfg = normalize_site_config(yaml.safe_load(stream), site_name)
|
cfg = normalize_site_config(yaml.safe_load(stream), site_name)
|
||||||
|
cfg["selected"] = _artifact_selection(cfg, os.environ.get("SITE_ARTIFACTS"))
|
||||||
except (ConfigError, yaml.YAMLError) as exc:
|
except (ConfigError, yaml.YAMLError) as exc:
|
||||||
die(str(exc))
|
die(str(exc))
|
||||||
print("Site config:")
|
print("Site config:")
|
||||||
print(f" domain: {cfg['domain']}")
|
print(f" domain: {cfg['domain']}")
|
||||||
print(f" contract: {cfg['compatibility'] or 'split-surface-v2'}")
|
print(f" contract: {cfg['compatibility'] or 'split-surface-v2'}")
|
||||||
print(f" artifacts: {[item['name'] for item in cfg['artifacts']]}")
|
print(f" artifacts: {[item['name'] for item in cfg['artifacts']]}")
|
||||||
|
print(f" publishing: {cfg['selected']}")
|
||||||
print(f" routes: {[(item['path'], item['access']) for item in cfg['routes']]}")
|
print(f" routes: {[(item['path'], item['access']) for item in cfg['routes']]}")
|
||||||
return cfg
|
return cfg
|
||||||
|
|
||||||
@@ -436,10 +581,7 @@ def clone_apps(token):
|
|||||||
apps_dir = Path("/tmp/apps-deploy")
|
apps_dir = Path("/tmp/apps-deploy")
|
||||||
if apps_dir.exists():
|
if apps_dir.exists():
|
||||||
shutil.rmtree(apps_dir)
|
shutil.rmtree(apps_dir)
|
||||||
clone_env = os.environ.copy()
|
clone_env = git_auth_env(token)
|
||||||
clone_env["CI_BOT_TOKEN"] = token
|
|
||||||
clone_env["GIT_ASKPASS"] = str(Path(__file__).with_name("git-askpass.sh"))
|
|
||||||
clone_env["GIT_TERMINAL_PROMPT"] = "0"
|
|
||||||
url = f"https://{user}@{GITEA_HOST}/{APPS_REPO}.git"
|
url = f"https://{user}@{GITEA_HOST}/{APPS_REPO}.git"
|
||||||
run(["git", "clone", "--depth", "1", url, str(apps_dir)],
|
run(["git", "clone", "--depth", "1", url, str(apps_dir)],
|
||||||
display=f"git clone --depth 1 https://{user}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}", env=clone_env)
|
display=f"git clone --depth 1 https://{user}@{GITEA_HOST}/{APPS_REPO}.git {apps_dir}", env=clone_env)
|
||||||
@@ -477,13 +619,23 @@ def render_templates(action_dir, template_vars, app_dir, manifests_dir):
|
|||||||
print(f" Rendered {out_name}.j2 -> {destination}")
|
print(f" Rendered {out_name}.j2 -> {destination}")
|
||||||
|
|
||||||
|
|
||||||
def commit_and_push(apps_dir, message):
|
def git_auth_env(token):
|
||||||
|
"""Return credential-safe Git authentication shared by clone and push."""
|
||||||
|
auth_env = os.environ.copy()
|
||||||
|
auth_env["CI_BOT_TOKEN"] = token
|
||||||
|
auth_env["GIT_ASKPASS"] = str(Path(__file__).with_name("git-askpass.sh"))
|
||||||
|
auth_env["GIT_TERMINAL_PROMPT"] = "0"
|
||||||
|
return auth_env
|
||||||
|
|
||||||
|
|
||||||
|
def commit_and_push(apps_dir, message, token=None):
|
||||||
run(["git", "-C", str(apps_dir), "add", "-A"])
|
run(["git", "-C", str(apps_dir), "add", "-A"])
|
||||||
result = subprocess.run(["git", "-C", str(apps_dir), "diff", "--cached", "--quiet"], check=False)
|
result = subprocess.run(["git", "-C", str(apps_dir), "diff", "--cached", "--quiet"], check=False)
|
||||||
if result.returncode == 0:
|
if result.returncode == 0:
|
||||||
print("No manifest changes to commit")
|
print("No manifest changes to commit")
|
||||||
return False
|
return False
|
||||||
run(["git", "-C", str(apps_dir), "commit", "-m", message])
|
run(["git", "-C", str(apps_dir), "commit", "-m", message])
|
||||||
run(["git", "-C", str(apps_dir), "push"])
|
push_env = git_auth_env(token) if token else None
|
||||||
|
run(["git", "-C", str(apps_dir), "push"], env=push_env)
|
||||||
print("Manifests pushed — ArgoCD will sync")
|
print("Manifests pushed — ArgoCD will sync")
|
||||||
return True
|
return True
|
||||||
|
|||||||
@@ -4,6 +4,13 @@ metadata:
|
|||||||
name: {{ route.resource_name }}
|
name: {{ route.resource_name }}
|
||||||
namespace: {{ namespace }}
|
namespace: {{ namespace }}
|
||||||
annotations:
|
annotations:
|
||||||
|
{%- if not compatibility %}
|
||||||
|
site-publish.fritzlab.net/artifact: {{ route.artifact | tojson }}
|
||||||
|
site-publish.fritzlab.net/access: {{ route.access | tojson }}
|
||||||
|
site-publish.fritzlab.net/bucket: {{ route.artifact_config.bucket | tojson }}
|
||||||
|
site-publish.fritzlab.net/immutable-paths: {{ route.immutable_paths_json | tojson }}
|
||||||
|
site-publish.fritzlab.net/route-path: {{ route.path | tojson }}
|
||||||
|
{%- endif %}
|
||||||
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% if route.access_middleware %},{{ route.access_middleware }}@file{% endif %}{% for m in route.middlewares %},{{ m }}@file{% endfor %}
|
traefik.ingress.kubernetes.io/router.middlewares: https-redirect@file,retry-upstream@file{% if route.access_middleware %},{{ route.access_middleware }}@file{% endif %}{% for m in route.middlewares %},{{ m }}@file{% endfor %}
|
||||||
spec:
|
spec:
|
||||||
ingressClassName: traefik
|
ingressClassName: traefik
|
||||||
|
|||||||
Vendored
+1
@@ -15,6 +15,7 @@ artifacts:
|
|||||||
- name: distributions
|
- name: distributions
|
||||||
type: static
|
type: static
|
||||||
content_dir: dist
|
content_dir: dist
|
||||||
|
cors_origins: ['*']
|
||||||
publish:
|
publish:
|
||||||
bucket: baseline-dist
|
bucket: baseline-dist
|
||||||
credentials:
|
credentials:
|
||||||
|
|||||||
+872
-13
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user