Decouple application releases from shared edge

This commit is contained in:
SimpleTest 2026-07-26 21:56:12 +03:00
parent 882df25aa9
commit 41011fe65c
14 changed files with 223 additions and 148 deletions

View File

@ -31,10 +31,14 @@ HTTP_BIND_ADDRESS=0.0.0.0
# network. Keep disabled for ordinary local development. # network. Keep disabled for ordinary local development.
PUBLIC_EDGE_ENABLED=false PUBLIC_EDGE_ENABLED=false
PUBLIC_EDGE_NETWORK=who_need_help_public_edge PUBLIC_EDGE_NETWORK=who_need_help_public_edge
PUBLIC_ROUTE_ID=who_need_help
PUBLIC_UPSTREAM_NAME=who-need-help-local PUBLIC_UPSTREAM_NAME=who-need-help-local
# The public Caddy edge is managed from the production checkout with the same PUBLIC_UPSTREAM_PORT=4000
# production .env. The test checkout joins PUBLIC_EDGE_NETWORK but never owns PUBLIC_HEALTH_PATH=/healthz/ready
# or restarts Caddy. PUBLIC_WWW_REDIRECT=false
# Legacy shared-edge settings remain while the submitted test deployment is
# frozen. New application releases do not build or restart Caddy. After the
# judging freeze, migrate these routes to the independent server_edge project.
EDGE_COMPOSE_PROJECT_NAME=who_need_help_edge EDGE_COMPOSE_PROJECT_NAME=who_need_help_edge
CADDY_IMAGE=who-need-help:caddy-local CADDY_IMAGE=who-need-help:caddy-local
EDGE_BIND_ADDRESS=0.0.0.0 EDGE_BIND_ADDRESS=0.0.0.0

View File

@ -106,8 +106,8 @@ policies are deliberately not claimed as complete.
## Fast start with Docker Compose ## Fast start with Docker Compose
The public single-server path uses the compact application topology plus a The public single-server path uses the compact application topology plus an
separately managed Caddy edge. Production and test can run as isolated independent server-level Caddy edge. Production and test can run as isolated
Compose projects with distinct PostGIS volumes and secrets while sharing only a Compose projects with distinct PostGIS volumes and secrets while sharing only a
Docker network used for HTTPS reverse proxying. See the Docker network used for HTTPS reverse proxying. See the
[operations runbook](docs/operations.md#two-independent-checkouts-and-one-env-in-each) [operations runbook](docs/operations.md#two-independent-checkouts-and-one-env-in-each)

View File

@ -88,7 +88,8 @@ The server uses exactly these independent Git clones:
/srv/who_need_help-production/.env /srv/who_need_help-production/.env
``` ```
No `.env.test`, `.env.staging`, `.env.production`, or `.env.edge` is used. No `.env.test`, `.env.staging`, or `.env.production` is used inside either
application checkout.
Each checkout can remain on a different commit. Test and production have Each checkout can remain on a different commit. Test and production have
different Compose projects, application/infrastructure image tags, database different Compose projects, application/infrastructure image tags, database
connections, Docker volumes, public aliases, Google OAuth clients, email connections, Docker volumes, public aliases, Google OAuth clients, email
@ -109,12 +110,19 @@ Those files are generated automatically, never copied to a server, and do not
represent dev, test, or production. The one ignored `.env` at each checkout represent dev, test, or production. The one ignored `.env` at each checkout
root remains the only application/deployment configuration. root remains the only application/deployment configuration.
The shared Caddy edge is owned only by the production checkout and reads the The current submitted deployment still has a legacy shared Caddy container
same production `.env`; it is not a third project directory or a second secret created from the production checkout. Keep it running and unchanged while the
file. Both applications intentionally share only the external judging test URL is frozen. Application release and rollback scripts do not
`who_need_help_public_edge` Docker network. Caddy routes `whoneedhelp.com` to build, recreate, or select an image for that container.
`who-need-help-production:4000` and `test.whoneedhelp.com` to
`who-need-help-test:4000`. The replacement is a separate server-level `server_edge` project with its own
single mode-`0600` `.env`, route directory, Caddy image, certificate volumes,
and Compose lifecycle. Each application keeps only its route contract in its
existing `.env`: `PHX_HOST`, `PUBLIC_ROUTE_ID`, `PUBLIC_EDGE_NETWORK`,
`PUBLIC_UPSTREAM_NAME`, `PUBLIC_UPSTREAM_PORT`, `PUBLIC_HEALTH_PATH`, and
`PUBLIC_WWW_REDIRECT`. No application deployment is allowed to restart the
server edge. Both applications intentionally share only the external public
edge Docker network.
### Environment-specific Android builds and App Links ### Environment-specific Android builds and App Links
@ -325,12 +333,15 @@ credentials:
``` ```
Start/update test first and run the complete browser/API/Android verification. Start/update test first and run the complete browser/API/Android verification.
Only then check out that exact tested SHA in production. Start the shared edge Only then check out that exact tested SHA in production. Start the application
from the production checkout with `./scripts/edge-up.sh .env`; start the with `./scripts/deploy-up.sh .env`. The application command joins the external
production application separately with `./scripts/deploy-up.sh .env`. public network but does not own or restart Caddy.
`edge-up.sh` explicitly selects `EDGE_COMPOSE_PROJECT_NAME`, so the shared
proxy remains independent from both application Compose projects even though The legacy `./scripts/edge-up.sh .env` command exists only for the currently
the same production `.env` also contains `COMPOSE_PROJECT_NAME`. frozen submitted deployment. Do not use it from an ordinary application
release. After judging, render and validate each route through the independent
`server_edge` workflow, transfer certificate-volume ownership in a reviewed
maintenance window, and only then retire the legacy edge container.
The authoritative A records for `whoneedhelp.com`, `www.whoneedhelp.com`, and The authoritative A records for `whoneedhelp.com`, `www.whoneedhelp.com`, and
`test.whoneedhelp.com` must point to the verified server address before Caddy `test.whoneedhelp.com` must point to the verified server address before Caddy
@ -953,9 +964,9 @@ The apply path refuses tracked local or remote modifications. It then:
backup again under local ignored `output/production-backups/`; backup again under local ignored `output/production-backups/`;
5. fast-forwards the production checkout without accessing or changing the 5. fast-forwards the production checkout without accessing or changing the
test checkout or public remote; test checkout or public remote;
6. selects immutable per-commit image tags, applies migrations, starts the 6. selects immutable per-commit application image tags, applies migrations,
application and edge, and verifies the public readiness and Android App starts only the application topology, and verifies public readiness through
Links endpoints. the already-running shared edge plus the Android App Links endpoints.
Every newly added migration must have one reviewed entry in Every newly added migration must have one reviewed entry in
`priv/repo/migration_application_compatibility.tsv`. `application_safe` means `priv/repo/migration_application_compatibility.tsv`. `application_safe` means
@ -977,8 +988,8 @@ and records that boundary in the release manifest. Restarting an older image
against a potentially incompatible schema is never automatic. against a potentially incompatible schema is never automatic.
For an `application_safe` release, an application startup failure restores the For an `application_safe` release, an application startup failure restores the
previous immutable application and Caddy image tags and attempts to recover previous immutable application image tags and attempts to recover public
public readiness. A `forward_only` release never starts the old application readiness through the unchanged edge. A `forward_only` release never starts the old application
after migration begins. Neither path reverses Git source or Ecto migrations after migration begins. Neither path reverses Git source or Ecto migrations
automatically. The per-release rollback manifest and backup paths are recorded automatically. The per-release rollback manifest and backup paths are recorded
below the production checkout's ignored `output/releases/`. The copied backup below the production checkout's ignored `output/releases/`. The copied backup
@ -1009,8 +1020,8 @@ image rollback is available only when the manifest records
`migration_policy=application_safe`. `migration_policy=application_safe`.
The plan requires the manifest target to be the currently checked-out The plan requires the manifest target to be the currently checked-out
production commit, verifies the previous immutable application and edge images production commit, verifies the previous immutable application image still
still exist, checks the pre-release backup catalog and checksum, and prints the exists, checks the pre-release backup catalog and checksum, and prints the
exact confirmation token. It does not change the remote environment or exact confirmation token. It does not change the remote environment or
containers. containers.
@ -1024,11 +1035,12 @@ WNH_PRODUCTION_ROLLBACK_CONFIRM=whoneedhelp.com:TARGET_COMMIT:PREVIOUS_COMMIT \
whoneedhelp whoneedhelp
``` ```
This application rollback atomically restores the four previous image This application rollback atomically restores the three previous application
selectors and recreates only the selected application topology and shared edge image selectors and recreates only the selected application topology with
with `--no-build`. It verifies the resulting image identities, container `--no-build`. It never changes or recreates the shared edge. It verifies the
health, public readiness, and App Links. A failed rollback attempts to restore resulting image identities, container health, public readiness through the
the pre-rollback image selection. The Git checkout intentionally remains at unchanged edge, and App Links. A failed rollback attempts to restore the
pre-rollback image selection. The Git checkout intentionally remains at
the newer source commit so the reviewed release tooling and manifest remain the newer source commit so the reviewed release tooling and manifest remain
available. available.

View File

@ -74,7 +74,11 @@ app_topology=${PRODUCTION_APP_TOPOLOGY:-compact}
compose_project_name=${PRODUCTION_COMPOSE_PROJECT_NAME:-who_need_help_production} compose_project_name=${PRODUCTION_COMPOSE_PROJECT_NAME:-who_need_help_production}
public_edge_enabled=${PRODUCTION_PUBLIC_EDGE_ENABLED:-true} public_edge_enabled=${PRODUCTION_PUBLIC_EDGE_ENABLED:-true}
public_edge_network=${PRODUCTION_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge} public_edge_network=${PRODUCTION_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge}
public_route_id=${PRODUCTION_PUBLIC_ROUTE_ID:-who_need_help_production}
public_upstream_name=${PRODUCTION_PUBLIC_UPSTREAM_NAME:-who-need-help-production} public_upstream_name=${PRODUCTION_PUBLIC_UPSTREAM_NAME:-who-need-help-production}
public_upstream_port=${PRODUCTION_PUBLIC_UPSTREAM_PORT:-4000}
public_health_path=${PRODUCTION_PUBLIC_HEALTH_PATH:-/healthz/ready}
public_www_redirect=${PRODUCTION_PUBLIC_WWW_REDIRECT:-true}
codex_session_id=${PRODUCTION_CODEX_SESSION_ID:-} codex_session_id=${PRODUCTION_CODEX_SESSION_ID:-}
google_oauth_client_id=${PRODUCTION_GOOGLE_OAUTH_CLIENT_ID:-} google_oauth_client_id=${PRODUCTION_GOOGLE_OAUTH_CLIENT_ID:-}
google_oauth_client_secret=${PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET:-} google_oauth_client_secret=${PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET:-}
@ -101,7 +105,11 @@ require_single_line_env_value PRODUCTION_APP_TOPOLOGY "$app_topology"
require_single_line_env_value PRODUCTION_COMPOSE_PROJECT_NAME "$compose_project_name" require_single_line_env_value PRODUCTION_COMPOSE_PROJECT_NAME "$compose_project_name"
require_single_line_env_value PRODUCTION_PUBLIC_EDGE_ENABLED "$public_edge_enabled" require_single_line_env_value PRODUCTION_PUBLIC_EDGE_ENABLED "$public_edge_enabled"
require_single_line_env_value PRODUCTION_PUBLIC_EDGE_NETWORK "$public_edge_network" require_single_line_env_value PRODUCTION_PUBLIC_EDGE_NETWORK "$public_edge_network"
require_single_line_env_value PRODUCTION_PUBLIC_ROUTE_ID "$public_route_id"
require_single_line_env_value PRODUCTION_PUBLIC_UPSTREAM_NAME "$public_upstream_name" require_single_line_env_value PRODUCTION_PUBLIC_UPSTREAM_NAME "$public_upstream_name"
require_single_line_env_value PRODUCTION_PUBLIC_UPSTREAM_PORT "$public_upstream_port"
require_single_line_env_value PRODUCTION_PUBLIC_HEALTH_PATH "$public_health_path"
require_single_line_env_value PRODUCTION_PUBLIC_WWW_REDIRECT "$public_www_redirect"
require_single_line_env_value PRODUCTION_CODEX_SESSION_ID "$codex_session_id" require_single_line_env_value PRODUCTION_CODEX_SESSION_ID "$codex_session_id"
require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_ID "$google_oauth_client_id" require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_ID "$google_oauth_client_id"
require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET "$google_oauth_client_secret" require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET "$google_oauth_client_secret"
@ -123,6 +131,30 @@ require_single_line_env_value PRODUCTION_TEST_DOMAIN "$test_domain"
require_single_line_env_value PRODUCTION_TEST_UPSTREAM "$test_upstream" require_single_line_env_value PRODUCTION_TEST_UPSTREAM "$test_upstream"
require_single_line_env_value PRODUCTION_EDGE_COMPOSE_PROJECT_NAME "$edge_compose_project_name" require_single_line_env_value PRODUCTION_EDGE_COMPOSE_PROJECT_NAME "$edge_compose_project_name"
if ! printf '%s\n' "$public_route_id" |
grep -Eq '^[a-z0-9][a-z0-9_-]{0,62}$'; then
echo "PRODUCTION_PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
fi
if ! printf '%s\n' "$public_upstream_port" | grep -Eq '^[0-9]+$' ||
[ "$public_upstream_port" -lt 1 ] ||
[ "$public_upstream_port" -gt 65535 ]; then
echo "PRODUCTION_PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
if ! printf '%s\n' "$public_health_path" |
grep -Eq '^/[A-Za-z0-9._~:@%/+,=-]*$'; then
echo "PRODUCTION_PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
fi
case "$public_www_redirect" in
true | false) ;;
*)
echo "PRODUCTION_PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
;;
esac
if [ -z "$codex_session_id" ]; then if [ -z "$codex_session_id" ]; then
echo "PRODUCTION_CODEX_SESSION_ID is required for the Build Week feedback page." >&2 echo "PRODUCTION_CODEX_SESSION_ID is required for the Build Week feedback page." >&2
exit 1 exit 1
@ -385,7 +417,11 @@ HTTP_BIND_ADDRESS_VALUE=$http_bind_address \
HTTP_PORT_VALUE=$http_port \ HTTP_PORT_VALUE=$http_port \
PUBLIC_EDGE_ENABLED_VALUE=$public_edge_enabled \ PUBLIC_EDGE_ENABLED_VALUE=$public_edge_enabled \
PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \ PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \
PUBLIC_ROUTE_ID_VALUE=$public_route_id \
PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \ PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \
PUBLIC_UPSTREAM_PORT_VALUE=$public_upstream_port \
PUBLIC_HEALTH_PATH_VALUE=$public_health_path \
PUBLIC_WWW_REDIRECT_VALUE=$public_www_redirect \
DOCKER_SOCKET_GID_VALUE=$docker_socket_gid \ DOCKER_SOCKET_GID_VALUE=$docker_socket_gid \
TRUSTED_PROXY_IPS_VALUE=$trusted_proxy_ips \ TRUSTED_PROXY_IPS_VALUE=$trusted_proxy_ips \
POSTGRES_PASSWORD_VALUE=$postgres_password \ POSTGRES_PASSWORD_VALUE=$postgres_password \
@ -423,7 +459,7 @@ ANDROID_APP_LINKS_PACKAGE_NAME_VALUE=$android_app_links_package_name \
ANDROID_APP_LINKS_FINGERPRINTS_VALUE=$android_app_links_fingerprints \ ANDROID_APP_LINKS_FINGERPRINTS_VALUE=$android_app_links_fingerprints \
ANDROID_PLAY_APP_SIGNING_FINGERPRINTS_VALUE=$android_play_app_signing_fingerprints \ ANDROID_PLAY_APP_SIGNING_FINGERPRINTS_VALUE=$android_play_app_signing_fingerprints \
EDGE_COMPOSE_PROJECT_NAME_VALUE=$edge_compose_project_name \ EDGE_COMPOSE_PROJECT_NAME_VALUE=$edge_compose_project_name \
PRIMARY_UPSTREAM_VALUE="$public_upstream_name:4000" \ PRIMARY_UPSTREAM_VALUE="$public_upstream_name:$public_upstream_port" \
TEST_DOMAIN_VALUE=$test_domain \ TEST_DOMAIN_VALUE=$test_domain \
TEST_UPSTREAM_VALUE=$test_upstream \ TEST_UPSTREAM_VALUE=$test_upstream \
awk ' awk '
@ -440,7 +476,11 @@ TEST_UPSTREAM_VALUE=$test_upstream \
replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"] replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"]
replacement["PUBLIC_EDGE_ENABLED"] = ENVIRON["PUBLIC_EDGE_ENABLED_VALUE"] replacement["PUBLIC_EDGE_ENABLED"] = ENVIRON["PUBLIC_EDGE_ENABLED_VALUE"]
replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"] replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"]
replacement["PUBLIC_ROUTE_ID"] = ENVIRON["PUBLIC_ROUTE_ID_VALUE"]
replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"] replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"]
replacement["PUBLIC_UPSTREAM_PORT"] = ENVIRON["PUBLIC_UPSTREAM_PORT_VALUE"]
replacement["PUBLIC_HEALTH_PATH"] = ENVIRON["PUBLIC_HEALTH_PATH_VALUE"]
replacement["PUBLIC_WWW_REDIRECT"] = ENVIRON["PUBLIC_WWW_REDIRECT_VALUE"]
replacement["EDGE_COMPOSE_PROJECT_NAME"] = ENVIRON["EDGE_COMPOSE_PROJECT_NAME_VALUE"] replacement["EDGE_COMPOSE_PROJECT_NAME"] = ENVIRON["EDGE_COMPOSE_PROJECT_NAME_VALUE"]
replacement["CADDY_IMAGE"] = "who-need-help:caddy-production-" ENVIRON["GIT_SHA_VALUE"] replacement["CADDY_IMAGE"] = "who-need-help:caddy-production-" ENVIRON["GIT_SHA_VALUE"]
replacement["PRIMARY_DOMAIN"] = ENVIRON["DOMAIN"] replacement["PRIMARY_DOMAIN"] = ENVIRON["DOMAIN"]

View File

@ -66,7 +66,11 @@ target_dir=$(dirname -- "$target")
compose_project_name=${TEST_COMPOSE_PROJECT_NAME:-who_need_help_test} compose_project_name=${TEST_COMPOSE_PROJECT_NAME:-who_need_help_test}
public_edge_network=${TEST_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge} public_edge_network=${TEST_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge}
public_route_id=${TEST_PUBLIC_ROUTE_ID:-who_need_help_test}
public_upstream_name=${TEST_PUBLIC_UPSTREAM_NAME:-who-need-help-test} public_upstream_name=${TEST_PUBLIC_UPSTREAM_NAME:-who-need-help-test}
public_upstream_port=${TEST_PUBLIC_UPSTREAM_PORT:-4000}
public_health_path=${TEST_PUBLIC_HEALTH_PATH:-/healthz/ready}
public_www_redirect=${TEST_PUBLIC_WWW_REDIRECT:-false}
http_bind_address=${TEST_HTTP_BIND_ADDRESS:-127.0.0.1} http_bind_address=${TEST_HTTP_BIND_ADDRESS:-127.0.0.1}
http_port=${TEST_HTTP_PORT:-4011} http_port=${TEST_HTTP_PORT:-4011}
mailpit_bind_address=${TEST_MAILPIT_BIND_ADDRESS:-127.0.0.1} mailpit_bind_address=${TEST_MAILPIT_BIND_ADDRESS:-127.0.0.1}
@ -91,7 +95,11 @@ git_sha=$(git -C "$ROOT" rev-parse --short=12 HEAD)
require_single_line_env_value TEST_COMPOSE_PROJECT_NAME "$compose_project_name" require_single_line_env_value TEST_COMPOSE_PROJECT_NAME "$compose_project_name"
require_single_line_env_value TEST_PUBLIC_EDGE_NETWORK "$public_edge_network" require_single_line_env_value TEST_PUBLIC_EDGE_NETWORK "$public_edge_network"
require_single_line_env_value TEST_PUBLIC_ROUTE_ID "$public_route_id"
require_single_line_env_value TEST_PUBLIC_UPSTREAM_NAME "$public_upstream_name" require_single_line_env_value TEST_PUBLIC_UPSTREAM_NAME "$public_upstream_name"
require_single_line_env_value TEST_PUBLIC_UPSTREAM_PORT "$public_upstream_port"
require_single_line_env_value TEST_PUBLIC_HEALTH_PATH "$public_health_path"
require_single_line_env_value TEST_PUBLIC_WWW_REDIRECT "$public_www_redirect"
require_single_line_env_value TEST_HTTP_BIND_ADDRESS "$http_bind_address" require_single_line_env_value TEST_HTTP_BIND_ADDRESS "$http_bind_address"
require_single_line_env_value TEST_HTTP_PORT "$http_port" require_single_line_env_value TEST_HTTP_PORT "$http_port"
require_single_line_env_value TEST_MAILPIT_BIND_ADDRESS "$mailpit_bind_address" require_single_line_env_value TEST_MAILPIT_BIND_ADDRESS "$mailpit_bind_address"
@ -122,10 +130,27 @@ require_single_line_env_value TEST_SUPPORT_INBOX_ADDRESS "$support_inbox_address
echo "TEST_PUBLIC_EDGE_NETWORK contains unsupported characters." >&2 echo "TEST_PUBLIC_EDGE_NETWORK contains unsupported characters." >&2
exit 1 exit 1
} }
[[ "$public_route_id" =~ ^[a-z0-9][a-z0-9_-]{0,62}$ ]] || {
echo "TEST_PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
}
[[ "$public_upstream_name" =~ ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ ]] || { [[ "$public_upstream_name" =~ ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ ]] || {
echo "TEST_PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2 echo "TEST_PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2
exit 1 exit 1
} }
if ! [[ "$public_upstream_port" =~ ^[0-9]+$ ]] ||
((public_upstream_port < 1 || public_upstream_port > 65535)); then
echo "TEST_PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
[[ "$public_health_path" =~ ^/[A-Za-z0-9._~:@%/+,=-]*$ ]] || {
echo "TEST_PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
}
[[ "$public_www_redirect" == true || "$public_www_redirect" == false ]] || {
echo "TEST_PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
}
[[ -n "$codex_session_id" ]] || { [[ -n "$codex_session_id" ]] || {
echo "TEST_CODEX_SESSION_ID is required for the Build Week feedback page." >&2 echo "TEST_CODEX_SESSION_ID is required for the Build Week feedback page." >&2
exit 1 exit 1
@ -269,7 +294,11 @@ DOMAIN=$domain \
GIT_SHA_VALUE=$git_sha \ GIT_SHA_VALUE=$git_sha \
COMPOSE_PROJECT_NAME_VALUE=$compose_project_name \ COMPOSE_PROJECT_NAME_VALUE=$compose_project_name \
PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \ PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \
PUBLIC_ROUTE_ID_VALUE=$public_route_id \
PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \ PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \
PUBLIC_UPSTREAM_PORT_VALUE=$public_upstream_port \
PUBLIC_HEALTH_PATH_VALUE=$public_health_path \
PUBLIC_WWW_REDIRECT_VALUE=$public_www_redirect \
HTTP_BIND_ADDRESS_VALUE=$http_bind_address \ HTTP_BIND_ADDRESS_VALUE=$http_bind_address \
HTTP_PORT_VALUE=$http_port \ HTTP_PORT_VALUE=$http_port \
MAILPIT_BIND_ADDRESS_VALUE=$mailpit_bind_address \ MAILPIT_BIND_ADDRESS_VALUE=$mailpit_bind_address \
@ -312,7 +341,11 @@ CODEX_SESSION_ID_VALUE=$codex_session_id \
replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"] replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"]
replacement["PUBLIC_EDGE_ENABLED"] = "true" replacement["PUBLIC_EDGE_ENABLED"] = "true"
replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"] replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"]
replacement["PUBLIC_ROUTE_ID"] = ENVIRON["PUBLIC_ROUTE_ID_VALUE"]
replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"] replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"]
replacement["PUBLIC_UPSTREAM_PORT"] = ENVIRON["PUBLIC_UPSTREAM_PORT_VALUE"]
replacement["PUBLIC_HEALTH_PATH"] = ENVIRON["PUBLIC_HEALTH_PATH_VALUE"]
replacement["PUBLIC_WWW_REDIRECT"] = ENVIRON["PUBLIC_WWW_REDIRECT_VALUE"]
replacement["MAILPIT_BIND_ADDRESS"] = ENVIRON["MAILPIT_BIND_ADDRESS_VALUE"] replacement["MAILPIT_BIND_ADDRESS"] = ENVIRON["MAILPIT_BIND_ADDRESS_VALUE"]
replacement["MAILPIT_PORT"] = ENVIRON["MAILPIT_PORT_VALUE"] replacement["MAILPIT_PORT"] = ENVIRON["MAILPIT_PORT_VALUE"]
replacement["DOCKER_SOCKET_GID"] = ENVIRON["DOCKER_SOCKET_GID_VALUE"] replacement["DOCKER_SOCKET_GID"] = ENVIRON["DOCKER_SOCKET_GID_VALUE"]

View File

@ -71,7 +71,6 @@ sed -i \
-e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \ -e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \ -e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \ -e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^CADDY_IMAGE=.*|CADDY_IMAGE=who-need-help:caddy-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
"$env_file" "$env_file"
EOF EOF
@ -158,18 +157,6 @@ if [ "$1" = inspect ]; then
esac esac
exit 0 exit 0
fi fi
if [ "$1" = compose ]; then
case " $* " in
*' build edge '*)
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
*' up -d --no-deps --no-build --wait edge '*)
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
esac
fi
printf 'Unexpected docker invocation: %s\n' "$*" >&2 printf 'Unexpected docker invocation: %s\n' "$*" >&2
exit 1 exit 1
EOF EOF
@ -222,6 +209,8 @@ run_release() {
run_release forward_only >"$run_dir/forward-success.out" run_release forward_only >"$run_dir/forward-success.out"
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \ grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -F 'compose:build migrate' "$fixture/mock-commands.log" >/dev/null grep -F 'compose:build migrate' "$fixture/mock-commands.log" >/dev/null
grep -F 'compose:stop app' "$fixture/mock-commands.log" >/dev/null grep -F 'compose:stop app' "$fixture/mock-commands.log" >/dev/null
grep -F 'compose:run --rm --no-deps migrate' \ grep -F 'compose:run --rm --no-deps migrate' \
@ -232,6 +221,15 @@ grep -R -F 'migration_policy=forward_only' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null "$fixture/output/releases" --include rollback-manifest.txt >/dev/null
grep -R -F 'status=success' \ grep -R -F 'status=success' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null "$fixture/output/releases" --include rollback-manifest.txt >/dev/null
if grep -R -E '^CADDY_IMAGE=' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null; then
echo "Application release manifest unexpectedly captured the shared edge image." >&2
exit 1
fi
if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Application release drill touched the shared edge." >&2
exit 1
fi
write_old_env write_old_env
printf '%s\n' "$current_commit" >"$fixture/git-state" printf '%s\n' "$current_commit" >"$fixture/git-state"
@ -252,6 +250,8 @@ grep -F 'previous application will not be restarted' \
"$run_dir/forward-failure.out" >/dev/null "$run_dir/forward-failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \ grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \ test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 1 "$fixture/mock-commands.log")" = 1
grep -R -F 'status=forward-only-release-failed' \ grep -R -F 'status=forward-only-release-failed' \
@ -276,6 +276,8 @@ grep -F 'restoring the previous immutable image tags' \
"$run_dir/safe-failure.out" >/dev/null "$run_dir/safe-failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${current_commit:0:12}" \ grep -Fx "APP_IMAGE=who-need-help:production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \ test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 2 "$fixture/mock-commands.log")" = 2

View File

@ -207,7 +207,6 @@ rollback_manifest="$release_dir/rollback-manifest.txt"
printf 'APP_IMAGE=%s\n' "$(read_value APP_IMAGE)" printf 'APP_IMAGE=%s\n' "$(read_value APP_IMAGE)"
printf 'SOCKET_PROXY_IMAGE=%s\n' "$(read_value SOCKET_PROXY_IMAGE)" printf 'SOCKET_PROXY_IMAGE=%s\n' "$(read_value SOCKET_PROXY_IMAGE)"
printf 'POSTGIS_IMAGE=%s\n' "$(read_value POSTGIS_IMAGE)" printf 'POSTGIS_IMAGE=%s\n' "$(read_value POSTGIS_IMAGE)"
printf 'CADDY_IMAGE=%s\n' "$(read_value CADDY_IMAGE)"
printf 'migration_policy=%s\n' "$migration_policy" printf 'migration_policy=%s\n' "$migration_policy"
printf 'migration_details=%s\n' \ printf 'migration_details=%s\n' \
"$(awk -F= '$1 == "migration_versions" {print $2}' <<<"$migration_policy_output")" "$(awk -F= '$1 == "migration_versions" {print $2}' <<<"$migration_policy_output")"
@ -228,13 +227,11 @@ restore_image_revision() {
APP_IMAGE_VALUE=$(awk -F= '$1 == "APP_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \ APP_IMAGE_VALUE=$(awk -F= '$1 == "APP_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
SOCKET_PROXY_IMAGE_VALUE=$(awk -F= '$1 == "SOCKET_PROXY_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \ SOCKET_PROXY_IMAGE_VALUE=$(awk -F= '$1 == "SOCKET_PROXY_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
POSTGIS_IMAGE_VALUE=$(awk -F= '$1 == "POSTGIS_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \ POSTGIS_IMAGE_VALUE=$(awk -F= '$1 == "POSTGIS_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
CADDY_IMAGE_VALUE=$(awk -F= '$1 == "CADDY_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
awk ' awk '
BEGIN { BEGIN {
replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"] replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"]
replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"] replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"]
replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"] replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"]
replacement["CADDY_IMAGE"] = ENVIRON["CADDY_IMAGE_VALUE"]
} }
{ {
separator = index($0, "=") separator = index($0, "=")
@ -267,14 +264,6 @@ rollback_runtime() {
"$root/scripts/compose.sh" "$env_file" \ "$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${runtime_services[@]}" || true up -d --no-deps --no-build --wait "${runtime_services[@]}" || true
edge_project=$(read_value EDGE_COMPOSE_PROJECT_NAME)
docker compose \
--project-name "$edge_project" \
--project-directory "$root" \
--env-file "$env_file" \
--file "$root/compose.edge.yaml" \
up -d --no-deps --no-build --wait edge || true
curl --fail --silent --show-error --max-time 15 \ curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null || true "https://$expected_domain/healthz/ready" >/dev/null || true
{ {
@ -312,15 +301,6 @@ revision_changed=true
"$root/scripts/check-environment-readiness.sh" "$env_file" --require-release "$root/scripts/check-environment-readiness.sh" "$env_file" --require-release
"$root/scripts/compose.sh" "$env_file" build "${build_services[@]}" "$root/scripts/compose.sh" "$env_file" build "${build_services[@]}"
edge_project=$(read_value EDGE_COMPOSE_PROJECT_NAME)
edge_compose=(
docker compose
--project-name "$edge_project"
--project-directory "$root"
--env-file "$env_file"
--file "$root/compose.edge.yaml"
)
"${edge_compose[@]}" build edge
if [[ "$migration_policy" == "forward_only" ]]; then if [[ "$migration_policy" == "forward_only" ]]; then
echo "Stopping the old application before the forward-only migration boundary." echo "Stopping the old application before the forward-only migration boundary."
@ -332,7 +312,6 @@ migration_started=true
"$root/scripts/check-database.sh" "$env_file" "$root/scripts/check-database.sh" "$env_file"
"$root/scripts/compose.sh" "$env_file" \ "$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${runtime_services[@]}" up -d --no-deps --no-build --wait "${runtime_services[@]}"
"${edge_compose[@]}" up -d --no-deps --no-build --wait edge
COMPOSE_PROJECT_NAME="$compose_project" \ COMPOSE_PROJECT_NAME="$compose_project" \
"$root/scripts/verify-realtime-cluster.sh" compose "$root/scripts/verify-realtime-cluster.sh" compose

View File

@ -43,6 +43,19 @@ else
exit 2 exit 2
fi fi
legacy_edge_paths=(
compose.edge.yaml
deploy/caddy/Caddyfile
)
if ! git -C "$ROOT" diff --quiet \
"$remote_commit" "$local_commit" -- "${legacy_edge_paths[@]}"; then
echo "The application release contains shared edge routing changes." >&2
echo "The legacy edge serves both production and the frozen test domain." >&2
echo "Move the approved route change through the independent server-edge workflow." >&2
exit 2
fi
echo "Shared edge routing files are unchanged; application release will not manage Caddy."
migration_policy_output=$( migration_policy_output=$(
"$ROOT/scripts/release-migration-policy.sh" "$remote_commit" "$local_commit" "$ROOT/scripts/release-migration-policy.sh" "$remote_commit" "$local_commit"
) )

View File

@ -13,6 +13,7 @@ manifest="$remote_root/output/releases/release-1/rollback-manifest.txt"
target_commit=2222222222222222222222222222222222222222 target_commit=2222222222222222222222222222222222222222
previous_commit=1111111111111111111111111111111111111111 previous_commit=1111111111111111111111111111111111111111
confirmation="whoneedhelp.com:$target_commit:$previous_commit" confirmation="whoneedhelp.com:$target_commit:$previous_commit"
edge_image=who-need-help:caddy-production-frozen-edge
cleanup() { cleanup() {
trap - EXIT HUP INT TERM trap - EXIT HUP INT TERM
@ -39,7 +40,7 @@ printf '%s\n' \
"APP_IMAGE=who-need-help:production-${target_commit:0:12}" \ "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}" \ "SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}" \
"POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}" \ "POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}" \
"CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}" \ "CADDY_IMAGE=$edge_image" \
>"$fixture/.env" >"$fixture/.env"
backup="$fixture/output/backups/production/pre-release.dump" backup="$fixture/output/backups/production/pre-release.dump"
@ -56,7 +57,6 @@ printf '%s\n' \
"APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \ "APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \
"SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${previous_commit:0:12}" \ "SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${previous_commit:0:12}" \
"POSTGIS_IMAGE=who-need-help:postgis-production-${previous_commit:0:12}" \ "POSTGIS_IMAGE=who-need-help:postgis-production-${previous_commit:0:12}" \
"CADDY_IMAGE=who-need-help:caddy-production-${previous_commit:0:12}" \
'migration_policy=application_safe' \ 'migration_policy=application_safe' \
"database_backup=$remote_root/output/backups/production/pre-release.dump" \ "database_backup=$remote_root/output/backups/production/pre-release.dump" \
'status=started' \ 'status=started' \
@ -75,7 +75,15 @@ shift
case "$*" in case "$*" in
'config --quiet') exit 0 ;; 'config --quiet') exit 0 ;;
'ps -q app') printf 'app-1\n'; exit 0 ;; 'ps -q app') printf 'app-1\n'; exit 0 ;;
up\ *) printf 'compose:%s\n' "$*" >>"$MOCK_COMMAND_LOG"; exit 0 ;; up\ *)
printf 'compose:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
if [ "${MOCK_FAIL_APP_UP:-}" = once ] &&
[ ! -e "$MOCK_FAIL_APP_MARKER" ]; then
: >"$MOCK_FAIL_APP_MARKER"
exit 17
fi
exit 0
;;
esac esac
printf 'Unexpected compose invocation: %s\n' "$*" >&2 printf 'Unexpected compose invocation: %s\n' "$*" >&2
exit 1 exit 1
@ -111,7 +119,6 @@ if [ "$1" = inspect ]; then
'{{.Config.Image}}') '{{.Config.Image}}')
case "$container" in case "$container" in
app-1) awk -F= '$1 == "APP_IMAGE" {print substr($0, index($0, "=") + 1)}' "$MOCK_ENV_FILE" ;; app-1) awk -F= '$1 == "APP_IMAGE" {print substr($0, index($0, "=") + 1)}' "$MOCK_ENV_FILE" ;;
edge-1) awk -F= '$1 == "CADDY_IMAGE" {print substr($0, index($0, "=") + 1)}' "$MOCK_ENV_FILE" ;;
*) exit 1 ;; *) exit 1 ;;
esac esac
;; ;;
@ -119,20 +126,6 @@ if [ "$1" = inspect ]; then
esac esac
exit 0 exit 0
fi fi
if [ "$1" = compose ]; then
case " $* " in
*' ps -q edge ') printf 'edge-1\n'; exit 0 ;;
*' up -d --no-deps --no-build --wait edge ')
if [ "${MOCK_FAIL_EDGE_UP:-}" = once ] &&
[ ! -e "$MOCK_FAIL_EDGE_MARKER" ]; then
: >"$MOCK_FAIL_EDGE_MARKER"
exit 17
fi
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
esac
fi
printf 'Unexpected docker invocation: %s\n' "$*" >&2 printf 'Unexpected docker invocation: %s\n' "$*" >&2
exit 1 exit 1
EOF EOF
@ -193,11 +186,14 @@ docker run --rm \
grep -Fx "APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \ grep -Fx "APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${previous_commit:0:12}" \ grep -Fx "CADDY_IMAGE=$edge_image" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -F 'compose:up -d --no-deps --no-build --wait app' \ grep -F 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log" >/dev/null "$fixture/mock-commands.log" >/dev/null
grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Application rollback drill touched the shared edge." >&2
exit 1
fi
if grep -E -- 'migrate|--build|(^|[[:space:]])db([[:space:]]|$)' \ if grep -E -- 'migrate|--build|(^|[[:space:]])db([[:space:]]|$)' \
"$fixture/mock-commands.log" >/dev/null; then "$fixture/mock-commands.log" >/dev/null; then
echo "Rollback drill touched migrations, builds, or the database service." >&2 echo "Rollback drill touched migrations, builds, or the database service." >&2
@ -242,9 +238,9 @@ sed -i \
-e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${target_commit:0:12}|" \ -e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${target_commit:0:12}|" \
-e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}|" \ -e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}|" \
-e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}|" \ -e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}|" \
-e "s|^CADDY_IMAGE=.*|CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}|" \
"$fixture/.env" "$fixture/.env"
: >"$fixture/mock-commands.log" : >"$fixture/mock-commands.log"
rm -f "$fixture/mock-app-failed-once"
set +e set +e
docker run --rm \ docker run --rm \
@ -255,8 +251,8 @@ docker run --rm \
--cap-drop ALL \ --cap-drop ALL \
--security-opt no-new-privileges \ --security-opt no-new-privileges \
--env "WNH_PRODUCTION_ROLLBACK_CONFIRM=$confirmation" \ --env "WNH_PRODUCTION_ROLLBACK_CONFIRM=$confirmation" \
--env MOCK_FAIL_EDGE_UP=once \ --env MOCK_FAIL_APP_UP=once \
--env "MOCK_FAIL_EDGE_MARKER=$remote_root/mock-edge-failed-once" \ --env "MOCK_FAIL_APP_MARKER=$remote_root/mock-app-failed-once" \
"${container_env[@]}" \ "${container_env[@]}" \
"${container_mounts[@]}" \ "${container_mounts[@]}" \
"$BASE_IMAGE" \ "$BASE_IMAGE" \
@ -274,10 +270,13 @@ grep -F 'Rollback failed; restoring the pre-rollback image selection.' \
"$run_dir/failure.out" >/dev/null "$run_dir/failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \ grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}" \ grep -Fx "CADDY_IMAGE=$edge_image" \
"$fixture/.env" >/dev/null "$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \ test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 2 "$fixture/mock-commands.log")" = 2
grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Rollback recovery touched the shared edge." >&2
exit 1
fi
echo "Isolated production application rollback plan/apply/failure-recovery drill passed." echo "Isolated production application rollback plan/apply/failure-recovery drill passed."

View File

@ -86,7 +86,6 @@ database_mode=$(read_unique "$env_file" DATABASE_MODE)
app_topology=$(read_unique "$env_file" APP_TOPOLOGY) app_topology=$(read_unique "$env_file" APP_TOPOLOGY)
phx_host=$(read_unique "$env_file" PHX_HOST) phx_host=$(read_unique "$env_file" PHX_HOST)
public_origin=$(read_unique "$env_file" WNH_BASE_URL) public_origin=$(read_unique "$env_file" WNH_BASE_URL)
edge_project=$(read_unique "$env_file" EDGE_COMPOSE_PROJECT_NAME)
[[ "$deployment_environment" == production ]] || { [[ "$deployment_environment" == production ]] || {
echo "DEPLOYMENT_ENV is not production." >&2 echo "DEPLOYMENT_ENV is not production." >&2
@ -153,29 +152,23 @@ git -C "$root" merge-base --is-ancestor "$previous_commit" "$target_commit" || {
previous_app_image=$(read_unique "$manifest" APP_IMAGE) previous_app_image=$(read_unique "$manifest" APP_IMAGE)
previous_socket_image=$(read_unique "$manifest" SOCKET_PROXY_IMAGE) previous_socket_image=$(read_unique "$manifest" SOCKET_PROXY_IMAGE)
previous_postgis_image=$(read_unique "$manifest" POSTGIS_IMAGE) previous_postgis_image=$(read_unique "$manifest" POSTGIS_IMAGE)
previous_caddy_image=$(read_unique "$manifest" CADDY_IMAGE)
require_image "$previous_app_image" production- APP_IMAGE require_image "$previous_app_image" production- APP_IMAGE
require_image "$previous_socket_image" socket-proxy-production- SOCKET_PROXY_IMAGE require_image "$previous_socket_image" socket-proxy-production- SOCKET_PROXY_IMAGE
require_image "$previous_postgis_image" postgis-production- POSTGIS_IMAGE require_image "$previous_postgis_image" postgis-production- POSTGIS_IMAGE
require_image "$previous_caddy_image" caddy-production- CADDY_IMAGE
target_short=${target_commit:0:12} target_short=${target_commit:0:12}
current_app_image=$(read_unique "$env_file" APP_IMAGE) current_app_image=$(read_unique "$env_file" APP_IMAGE)
current_socket_image=$(read_unique "$env_file" SOCKET_PROXY_IMAGE) current_socket_image=$(read_unique "$env_file" SOCKET_PROXY_IMAGE)
current_postgis_image=$(read_unique "$env_file" POSTGIS_IMAGE) current_postgis_image=$(read_unique "$env_file" POSTGIS_IMAGE)
current_caddy_image=$(read_unique "$env_file" CADDY_IMAGE)
[[ "$current_app_image" == "who-need-help:production-$target_short" && [[ "$current_app_image" == "who-need-help:production-$target_short" &&
"$current_socket_image" == "who-need-help:socket-proxy-production-$target_short" && "$current_socket_image" == "who-need-help:socket-proxy-production-$target_short" &&
"$current_postgis_image" == "who-need-help:postgis-production-$target_short" && "$current_postgis_image" == "who-need-help:postgis-production-$target_short" ]] || {
"$current_caddy_image" == "who-need-help:caddy-production-$target_short" ]] || {
echo "Current production image selection does not match the manifest target commit." >&2 echo "Current production image selection does not match the manifest target commit." >&2
exit 2 exit 2
} }
for image in "$previous_app_image" "$previous_caddy_image"; do docker image inspect "$previous_app_image" >/dev/null
docker image inspect "$image" >/dev/null
done
backup=$(realpath --canonicalize-existing "$backup") backup=$(realpath --canonicalize-existing "$backup")
case "$backup" in case "$backup" in
@ -223,34 +216,7 @@ check_application() {
done done
} }
edge_compose=(
docker compose
--project-name "$edge_project"
--project-directory "$root"
--env-file "$env_file"
--file "$root/compose.edge.yaml"
)
check_edge() {
local expected_image=$1 container state health image
mapfile -t containers < <("${edge_compose[@]}" ps -q edge)
[[ ${#containers[@]} -gt 0 ]] || {
echo "Production edge service is not running." >&2
return 1
}
for container in "${containers[@]}"; do
state=$(docker inspect --format '{{.State.Status}}' "$container")
health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$container")
image=$(docker inspect --format '{{.Config.Image}}' "$container")
[[ "$state" == running && "$health" == healthy && "$image" == "$expected_image" ]] || {
echo "Production edge does not match the expected healthy image." >&2
return 1
}
done
}
check_application "$current_app_image" check_application "$current_app_image"
check_edge "$current_caddy_image"
curl --fail --silent --show-error --max-time 15 \ curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null "https://$expected_domain/healthz/ready" >/dev/null
@ -264,8 +230,8 @@ printf 'Database mode: %s (no restore or migration reversal)\n' "$database_mode"
printf 'Rollback manifest: %s\n' "$manifest" printf 'Rollback manifest: %s\n' "$manifest"
printf 'Verified backup evidence: %s\n' "$backup" printf 'Verified backup evidence: %s\n' "$backup"
printf 'Exact confirmation: %s\n' "$confirmation" printf 'Exact confirmation: %s\n' "$confirmation"
echo "Scope: update four image selectors in production .env; recreate only application and edge containers." echo "Scope: update three application image selectors in production .env; recreate only application containers."
echo "Excluded: Git checkout, database, migrations, test deployment, public Git, and Devpost." echo "Excluded: shared edge/Caddy, Git checkout, database, migrations, test deployment, public Git, and Devpost."
if [[ "$action" == plan ]]; then if [[ "$action" == plan ]]; then
echo "Read-only production application rollback scope check passed." echo "Read-only production application rollback scope check passed."
@ -278,20 +244,18 @@ if [[ "${WNH_PRODUCTION_ROLLBACK_CONFIRM:-}" != "$confirmation" ]]; then
fi fi
update_images() { update_images() {
local app_image=$1 socket_image=$2 postgis_image=$3 caddy_image=$4 temporary local app_image=$1 socket_image=$2 postgis_image=$3 temporary
temporary=$(mktemp "$root/.env.image-selection.XXXXXX") temporary=$(mktemp "$root/.env.image-selection.XXXXXX")
chmod 600 "$temporary" chmod 600 "$temporary"
APP_IMAGE_VALUE=$app_image \ APP_IMAGE_VALUE=$app_image \
SOCKET_PROXY_IMAGE_VALUE=$socket_image \ SOCKET_PROXY_IMAGE_VALUE=$socket_image \
POSTGIS_IMAGE_VALUE=$postgis_image \ POSTGIS_IMAGE_VALUE=$postgis_image \
CADDY_IMAGE_VALUE=$caddy_image \
awk ' awk '
BEGIN { BEGIN {
replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"] replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"]
replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"] replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"]
replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"] replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"]
replacement["CADDY_IMAGE"] = ENVIRON["CADDY_IMAGE_VALUE"]
} }
{ {
separator = index($0, "=") separator = index($0, "=")
@ -326,12 +290,9 @@ recover_current_runtime() {
update_images \ update_images \
"$current_app_image" \ "$current_app_image" \
"$current_socket_image" \ "$current_socket_image" \
"$current_postgis_image" \ "$current_postgis_image" || true
"$current_caddy_image" || true
"$root/scripts/compose.sh" "$env_file" \ "$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${app_services[@]}" || true up -d --no-deps --no-build --wait "${app_services[@]}" || true
"${edge_compose[@]}" \
up -d --no-deps --no-build --wait edge || true
curl --fail --silent --show-error --max-time 15 \ curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null || true "https://$expected_domain/healthz/ready" >/dev/null || true
fi fi
@ -342,17 +303,13 @@ trap recover_current_runtime EXIT HUP INT TERM
update_images \ update_images \
"$previous_app_image" \ "$previous_app_image" \
"$previous_socket_image" \ "$previous_socket_image" \
"$previous_postgis_image" \ "$previous_postgis_image"
"$previous_caddy_image"
runtime_changed=true runtime_changed=true
"$root/scripts/compose.sh" "$env_file" \ "$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${app_services[@]}" up -d --no-deps --no-build --wait "${app_services[@]}"
"${edge_compose[@]}" \
up -d --no-deps --no-build --wait edge
check_application "$previous_app_image" check_application "$previous_app_image"
check_edge "$previous_caddy_image"
curl --fail --silent --show-error --max-time 30 \ curl --fail --silent --show-error --max-time 30 \
"https://$expected_domain/healthz/ready" >/dev/null "https://$expected_domain/healthz/ready" >/dev/null
curl --fail --silent --show-error --max-time 30 \ curl --fail --silent --show-error --max-time 30 \

View File

@ -57,7 +57,7 @@ case "$deployment_env" in
required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE' required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE'
;; ;;
production) production)
required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE CADDY_IMAGE' required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE'
;; ;;
*) *)
echo "DEPLOYMENT_ENV must be test or production." >&2 echo "DEPLOYMENT_ENV must be test or production." >&2
@ -81,9 +81,6 @@ DEPLOYMENT_ENV_VALUE=$deployment_env GIT_SHA_VALUE=$git_sha awk '
replacement["APP_IMAGE"] = "who-need-help:" prefix "-" sha replacement["APP_IMAGE"] = "who-need-help:" prefix "-" sha
replacement["SOCKET_PROXY_IMAGE"] = "who-need-help:socket-proxy-" prefix "-" sha replacement["SOCKET_PROXY_IMAGE"] = "who-need-help:socket-proxy-" prefix "-" sha
replacement["POSTGIS_IMAGE"] = "who-need-help:postgis-" prefix "-" sha replacement["POSTGIS_IMAGE"] = "who-need-help:postgis-" prefix "-" sha
if (prefix == "production") {
replacement["CADDY_IMAGE"] = "who-need-help:caddy-production-" sha
}
} }
{ {
separator = index($0, "=") separator = index($0, "=")

View File

@ -79,7 +79,7 @@ require_different() {
} }
for key in COMPOSE_PROJECT_NAME APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE \ for key in COMPOSE_PROJECT_NAME APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE \
PHX_HOST WNH_BASE_URL PUBLIC_UPSTREAM_NAME DATABASE_URL EMAIL_FROM_ADDRESS \ PHX_HOST WNH_BASE_URL PUBLIC_ROUTE_ID PUBLIC_UPSTREAM_NAME DATABASE_URL EMAIL_FROM_ADDRESS \
SECRET_KEY_BASE HANDOVER_SECRET RELEASE_COOKIE METRICS_TOKEN; do SECRET_KEY_BASE HANDOVER_SECRET RELEASE_COOKIE METRICS_TOKEN; do
require_different "$key" require_different "$key"
done done
@ -133,14 +133,16 @@ production_edge_network=$(read_env "$production_env" PUBLIC_EDGE_NETWORK)
exit 1 exit 1
} }
test_upstream=$(read_env "$test_env" PUBLIC_UPSTREAM_NAME) test_upstream=$(read_env "$test_env" PUBLIC_UPSTREAM_NAME)
test_upstream_port=$(read_env "$test_env" PUBLIC_UPSTREAM_PORT)
production_edge_test_upstream=$(read_env "$production_env" TEST_UPSTREAM) production_edge_test_upstream=$(read_env "$production_env" TEST_UPSTREAM)
[[ "$production_edge_test_upstream" == "$test_upstream:4000" ]] || { [[ "$production_edge_test_upstream" == "$test_upstream:$test_upstream_port" ]] || {
echo "Production edge TEST_UPSTREAM does not point to the test alias." >&2 echo "Production edge TEST_UPSTREAM does not point to the test alias." >&2
exit 1 exit 1
} }
production_upstream=$(read_env "$production_env" PUBLIC_UPSTREAM_NAME) production_upstream=$(read_env "$production_env" PUBLIC_UPSTREAM_NAME)
production_upstream_port=$(read_env "$production_env" PUBLIC_UPSTREAM_PORT)
production_edge_primary_upstream=$(read_env "$production_env" PRIMARY_UPSTREAM) production_edge_primary_upstream=$(read_env "$production_env" PRIMARY_UPSTREAM)
[[ "$production_edge_primary_upstream" == "$production_upstream:4000" ]] || { [[ "$production_edge_primary_upstream" == "$production_upstream:$production_upstream_port" ]] || {
echo "Production edge PRIMARY_UPSTREAM does not point to production." >&2 echo "Production edge PRIMARY_UPSTREAM does not point to production." >&2
exit 1 exit 1
} }

View File

@ -109,7 +109,11 @@ debug_base_url=$(require_value WNH_DEBUG_BASE_URL)
http_bind_address=$(require_value HTTP_BIND_ADDRESS) http_bind_address=$(require_value HTTP_BIND_ADDRESS)
public_edge_enabled=$(require_value PUBLIC_EDGE_ENABLED) public_edge_enabled=$(require_value PUBLIC_EDGE_ENABLED)
public_edge_network=$(require_value PUBLIC_EDGE_NETWORK) public_edge_network=$(require_value PUBLIC_EDGE_NETWORK)
public_route_id=$(require_value PUBLIC_ROUTE_ID)
public_upstream_name=$(require_value PUBLIC_UPSTREAM_NAME) public_upstream_name=$(require_value PUBLIC_UPSTREAM_NAME)
public_upstream_port=$(require_value PUBLIC_UPSTREAM_PORT)
public_health_path=$(require_value PUBLIC_HEALTH_PATH)
public_www_redirect=$(require_value PUBLIC_WWW_REDIRECT)
trusted_proxy_ips=$(optional_value TRAEFIK_TRUSTED_IPS) trusted_proxy_ips=$(optional_value TRAEFIK_TRUSTED_IPS)
postgres_password=$(optional_value POSTGRES_PASSWORD) postgres_password=$(optional_value POSTGRES_PASSWORD)
postgres_db=$(optional_value POSTGRES_DB) postgres_db=$(optional_value POSTGRES_DB)
@ -203,6 +207,23 @@ if [[ "$public_edge_enabled" == true ]]; then
echo "PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2 echo "PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2
exit 1 exit 1
} }
[[ "$public_route_id" =~ ^[a-z0-9][a-z0-9_-]{0,62}$ ]] || {
echo "PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
}
if ! [[ "$public_upstream_port" =~ ^[0-9]+$ ]] ||
((public_upstream_port < 1 || public_upstream_port > 65535)); then
echo "PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
[[ "$public_health_path" =~ ^/[A-Za-z0-9._~:@%/+,=-]*$ ]] || {
echo "PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
}
[[ "$public_www_redirect" == true || "$public_www_redirect" == false ]] || {
echo "PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
}
fi fi
[[ "$phx_host" == "$expected_domain" ]] || { [[ "$phx_host" == "$expected_domain" ]] || {
@ -557,7 +578,7 @@ reject_marker CODEX_SESSION_ID "$codex_session_id"
echo "PRIMARY_DOMAIN does not match EXPECTED_DOMAIN." >&2 echo "PRIMARY_DOMAIN does not match EXPECTED_DOMAIN." >&2
exit 1 exit 1
} }
[[ "$primary_upstream" == "$(require_value PUBLIC_UPSTREAM_NAME):4000" ]] || { [[ "$primary_upstream" == "$public_upstream_name:$public_upstream_port" ]] || {
echo "PRIMARY_UPSTREAM does not target the production application alias." >&2 echo "PRIMARY_UPSTREAM does not target the production application alias." >&2
exit 1 exit 1
} }

View File

@ -136,6 +136,22 @@ require_value EMAIL_FROM_ADDRESS >/dev/null
echo "The test public upstream alias must be who-need-help-test." >&2 echo "The test public upstream alias must be who-need-help-test." >&2
exit 1 exit 1
} }
[[ "$(require_value PUBLIC_ROUTE_ID)" == who_need_help_test ]] || {
echo "The test route ID must be who_need_help_test." >&2
exit 1
}
[[ "$(require_value PUBLIC_UPSTREAM_PORT)" == 4000 ]] || {
echo "The test public upstream port must be 4000." >&2
exit 1
}
[[ "$(require_value PUBLIC_HEALTH_PATH)" == /healthz/ready ]] || {
echo "The test public health path must be /healthz/ready." >&2
exit 1
}
[[ "$(require_value PUBLIC_WWW_REDIRECT)" == false ]] || {
echo "The test route must not claim the primary www hostname." >&2
exit 1
}
google_id=$(read_value GOOGLE_OAUTH_CLIENT_ID 2>/dev/null || true) google_id=$(read_value GOOGLE_OAUTH_CLIENT_ID 2>/dev/null || true)
google_secret=$(read_value GOOGLE_OAUTH_CLIENT_SECRET 2>/dev/null || true) google_secret=$(read_value GOOGLE_OAUTH_CLIENT_SECRET 2>/dev/null || true)