Decouple application releases from shared edge

This commit is contained in:
SimpleTest 2026-07-26 21:56:12 +03:00
parent 882df25aa9
commit 41011fe65c
14 changed files with 223 additions and 148 deletions

View File

@ -31,10 +31,14 @@ HTTP_BIND_ADDRESS=0.0.0.0
# network. Keep disabled for ordinary local development.
PUBLIC_EDGE_ENABLED=false
PUBLIC_EDGE_NETWORK=who_need_help_public_edge
PUBLIC_ROUTE_ID=who_need_help
PUBLIC_UPSTREAM_NAME=who-need-help-local
# The public Caddy edge is managed from the production checkout with the same
# production .env. The test checkout joins PUBLIC_EDGE_NETWORK but never owns
# or restarts Caddy.
PUBLIC_UPSTREAM_PORT=4000
PUBLIC_HEALTH_PATH=/healthz/ready
PUBLIC_WWW_REDIRECT=false
# Legacy shared-edge settings remain while the submitted test deployment is
# frozen. New application releases do not build or restart Caddy. After the
# judging freeze, migrate these routes to the independent server_edge project.
EDGE_COMPOSE_PROJECT_NAME=who_need_help_edge
CADDY_IMAGE=who-need-help:caddy-local
EDGE_BIND_ADDRESS=0.0.0.0

View File

@ -106,8 +106,8 @@ policies are deliberately not claimed as complete.
## Fast start with Docker Compose
The public single-server path uses the compact application topology plus a
separately managed Caddy edge. Production and test can run as isolated
The public single-server path uses the compact application topology plus an
independent server-level Caddy edge. Production and test can run as isolated
Compose projects with distinct PostGIS volumes and secrets while sharing only a
Docker network used for HTTPS reverse proxying. See the
[operations runbook](docs/operations.md#two-independent-checkouts-and-one-env-in-each)

View File

@ -88,7 +88,8 @@ The server uses exactly these independent Git clones:
/srv/who_need_help-production/.env
```
No `.env.test`, `.env.staging`, `.env.production`, or `.env.edge` is used.
No `.env.test`, `.env.staging`, or `.env.production` is used inside either
application checkout.
Each checkout can remain on a different commit. Test and production have
different Compose projects, application/infrastructure image tags, database
connections, Docker volumes, public aliases, Google OAuth clients, email
@ -109,12 +110,19 @@ Those files are generated automatically, never copied to a server, and do not
represent dev, test, or production. The one ignored `.env` at each checkout
root remains the only application/deployment configuration.
The shared Caddy edge is owned only by the production checkout and reads the
same production `.env`; it is not a third project directory or a second secret
file. Both applications intentionally share only the external
`who_need_help_public_edge` Docker network. Caddy routes `whoneedhelp.com` to
`who-need-help-production:4000` and `test.whoneedhelp.com` to
`who-need-help-test:4000`.
The current submitted deployment still has a legacy shared Caddy container
created from the production checkout. Keep it running and unchanged while the
judging test URL is frozen. Application release and rollback scripts do not
build, recreate, or select an image for that container.
The replacement is a separate server-level `server_edge` project with its own
single mode-`0600` `.env`, route directory, Caddy image, certificate volumes,
and Compose lifecycle. Each application keeps only its route contract in its
existing `.env`: `PHX_HOST`, `PUBLIC_ROUTE_ID`, `PUBLIC_EDGE_NETWORK`,
`PUBLIC_UPSTREAM_NAME`, `PUBLIC_UPSTREAM_PORT`, `PUBLIC_HEALTH_PATH`, and
`PUBLIC_WWW_REDIRECT`. No application deployment is allowed to restart the
server edge. Both applications intentionally share only the external public
edge Docker network.
### Environment-specific Android builds and App Links
@ -325,12 +333,15 @@ credentials:
```
Start/update test first and run the complete browser/API/Android verification.
Only then check out that exact tested SHA in production. Start the shared edge
from the production checkout with `./scripts/edge-up.sh .env`; start the
production application separately with `./scripts/deploy-up.sh .env`.
`edge-up.sh` explicitly selects `EDGE_COMPOSE_PROJECT_NAME`, so the shared
proxy remains independent from both application Compose projects even though
the same production `.env` also contains `COMPOSE_PROJECT_NAME`.
Only then check out that exact tested SHA in production. Start the application
with `./scripts/deploy-up.sh .env`. The application command joins the external
public network but does not own or restart Caddy.
The legacy `./scripts/edge-up.sh .env` command exists only for the currently
frozen submitted deployment. Do not use it from an ordinary application
release. After judging, render and validate each route through the independent
`server_edge` workflow, transfer certificate-volume ownership in a reviewed
maintenance window, and only then retire the legacy edge container.
The authoritative A records for `whoneedhelp.com`, `www.whoneedhelp.com`, and
`test.whoneedhelp.com` must point to the verified server address before Caddy
@ -953,9 +964,9 @@ The apply path refuses tracked local or remote modifications. It then:
backup again under local ignored `output/production-backups/`;
5. fast-forwards the production checkout without accessing or changing the
test checkout or public remote;
6. selects immutable per-commit image tags, applies migrations, starts the
application and edge, and verifies the public readiness and Android App
Links endpoints.
6. selects immutable per-commit application image tags, applies migrations,
starts only the application topology, and verifies public readiness through
the already-running shared edge plus the Android App Links endpoints.
Every newly added migration must have one reviewed entry in
`priv/repo/migration_application_compatibility.tsv`. `application_safe` means
@ -977,8 +988,8 @@ and records that boundary in the release manifest. Restarting an older image
against a potentially incompatible schema is never automatic.
For an `application_safe` release, an application startup failure restores the
previous immutable application and Caddy image tags and attempts to recover
public readiness. A `forward_only` release never starts the old application
previous immutable application image tags and attempts to recover public
readiness through the unchanged edge. A `forward_only` release never starts the old application
after migration begins. Neither path reverses Git source or Ecto migrations
automatically. The per-release rollback manifest and backup paths are recorded
below the production checkout's ignored `output/releases/`. The copied backup
@ -1009,8 +1020,8 @@ image rollback is available only when the manifest records
`migration_policy=application_safe`.
The plan requires the manifest target to be the currently checked-out
production commit, verifies the previous immutable application and edge images
still exist, checks the pre-release backup catalog and checksum, and prints the
production commit, verifies the previous immutable application image still
exists, checks the pre-release backup catalog and checksum, and prints the
exact confirmation token. It does not change the remote environment or
containers.
@ -1024,11 +1035,12 @@ WNH_PRODUCTION_ROLLBACK_CONFIRM=whoneedhelp.com:TARGET_COMMIT:PREVIOUS_COMMIT \
whoneedhelp
```
This application rollback atomically restores the four previous image
selectors and recreates only the selected application topology and shared edge
with `--no-build`. It verifies the resulting image identities, container
health, public readiness, and App Links. A failed rollback attempts to restore
the pre-rollback image selection. The Git checkout intentionally remains at
This application rollback atomically restores the three previous application
image selectors and recreates only the selected application topology with
`--no-build`. It never changes or recreates the shared edge. It verifies the
resulting image identities, container health, public readiness through the
unchanged edge, and App Links. A failed rollback attempts to restore the
pre-rollback image selection. The Git checkout intentionally remains at
the newer source commit so the reviewed release tooling and manifest remain
available.

View File

@ -74,7 +74,11 @@ app_topology=${PRODUCTION_APP_TOPOLOGY:-compact}
compose_project_name=${PRODUCTION_COMPOSE_PROJECT_NAME:-who_need_help_production}
public_edge_enabled=${PRODUCTION_PUBLIC_EDGE_ENABLED:-true}
public_edge_network=${PRODUCTION_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge}
public_route_id=${PRODUCTION_PUBLIC_ROUTE_ID:-who_need_help_production}
public_upstream_name=${PRODUCTION_PUBLIC_UPSTREAM_NAME:-who-need-help-production}
public_upstream_port=${PRODUCTION_PUBLIC_UPSTREAM_PORT:-4000}
public_health_path=${PRODUCTION_PUBLIC_HEALTH_PATH:-/healthz/ready}
public_www_redirect=${PRODUCTION_PUBLIC_WWW_REDIRECT:-true}
codex_session_id=${PRODUCTION_CODEX_SESSION_ID:-}
google_oauth_client_id=${PRODUCTION_GOOGLE_OAUTH_CLIENT_ID:-}
google_oauth_client_secret=${PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET:-}
@ -101,7 +105,11 @@ require_single_line_env_value PRODUCTION_APP_TOPOLOGY "$app_topology"
require_single_line_env_value PRODUCTION_COMPOSE_PROJECT_NAME "$compose_project_name"
require_single_line_env_value PRODUCTION_PUBLIC_EDGE_ENABLED "$public_edge_enabled"
require_single_line_env_value PRODUCTION_PUBLIC_EDGE_NETWORK "$public_edge_network"
require_single_line_env_value PRODUCTION_PUBLIC_ROUTE_ID "$public_route_id"
require_single_line_env_value PRODUCTION_PUBLIC_UPSTREAM_NAME "$public_upstream_name"
require_single_line_env_value PRODUCTION_PUBLIC_UPSTREAM_PORT "$public_upstream_port"
require_single_line_env_value PRODUCTION_PUBLIC_HEALTH_PATH "$public_health_path"
require_single_line_env_value PRODUCTION_PUBLIC_WWW_REDIRECT "$public_www_redirect"
require_single_line_env_value PRODUCTION_CODEX_SESSION_ID "$codex_session_id"
require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_ID "$google_oauth_client_id"
require_single_line_env_value PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET "$google_oauth_client_secret"
@ -123,6 +131,30 @@ require_single_line_env_value PRODUCTION_TEST_DOMAIN "$test_domain"
require_single_line_env_value PRODUCTION_TEST_UPSTREAM "$test_upstream"
require_single_line_env_value PRODUCTION_EDGE_COMPOSE_PROJECT_NAME "$edge_compose_project_name"
if ! printf '%s\n' "$public_route_id" |
grep -Eq '^[a-z0-9][a-z0-9_-]{0,62}$'; then
echo "PRODUCTION_PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
fi
if ! printf '%s\n' "$public_upstream_port" | grep -Eq '^[0-9]+$' ||
[ "$public_upstream_port" -lt 1 ] ||
[ "$public_upstream_port" -gt 65535 ]; then
echo "PRODUCTION_PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
if ! printf '%s\n' "$public_health_path" |
grep -Eq '^/[A-Za-z0-9._~:@%/+,=-]*$'; then
echo "PRODUCTION_PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
fi
case "$public_www_redirect" in
true | false) ;;
*)
echo "PRODUCTION_PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
;;
esac
if [ -z "$codex_session_id" ]; then
echo "PRODUCTION_CODEX_SESSION_ID is required for the Build Week feedback page." >&2
exit 1
@ -385,7 +417,11 @@ HTTP_BIND_ADDRESS_VALUE=$http_bind_address \
HTTP_PORT_VALUE=$http_port \
PUBLIC_EDGE_ENABLED_VALUE=$public_edge_enabled \
PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \
PUBLIC_ROUTE_ID_VALUE=$public_route_id \
PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \
PUBLIC_UPSTREAM_PORT_VALUE=$public_upstream_port \
PUBLIC_HEALTH_PATH_VALUE=$public_health_path \
PUBLIC_WWW_REDIRECT_VALUE=$public_www_redirect \
DOCKER_SOCKET_GID_VALUE=$docker_socket_gid \
TRUSTED_PROXY_IPS_VALUE=$trusted_proxy_ips \
POSTGRES_PASSWORD_VALUE=$postgres_password \
@ -423,7 +459,7 @@ ANDROID_APP_LINKS_PACKAGE_NAME_VALUE=$android_app_links_package_name \
ANDROID_APP_LINKS_FINGERPRINTS_VALUE=$android_app_links_fingerprints \
ANDROID_PLAY_APP_SIGNING_FINGERPRINTS_VALUE=$android_play_app_signing_fingerprints \
EDGE_COMPOSE_PROJECT_NAME_VALUE=$edge_compose_project_name \
PRIMARY_UPSTREAM_VALUE="$public_upstream_name:4000" \
PRIMARY_UPSTREAM_VALUE="$public_upstream_name:$public_upstream_port" \
TEST_DOMAIN_VALUE=$test_domain \
TEST_UPSTREAM_VALUE=$test_upstream \
awk '
@ -440,7 +476,11 @@ TEST_UPSTREAM_VALUE=$test_upstream \
replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"]
replacement["PUBLIC_EDGE_ENABLED"] = ENVIRON["PUBLIC_EDGE_ENABLED_VALUE"]
replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"]
replacement["PUBLIC_ROUTE_ID"] = ENVIRON["PUBLIC_ROUTE_ID_VALUE"]
replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"]
replacement["PUBLIC_UPSTREAM_PORT"] = ENVIRON["PUBLIC_UPSTREAM_PORT_VALUE"]
replacement["PUBLIC_HEALTH_PATH"] = ENVIRON["PUBLIC_HEALTH_PATH_VALUE"]
replacement["PUBLIC_WWW_REDIRECT"] = ENVIRON["PUBLIC_WWW_REDIRECT_VALUE"]
replacement["EDGE_COMPOSE_PROJECT_NAME"] = ENVIRON["EDGE_COMPOSE_PROJECT_NAME_VALUE"]
replacement["CADDY_IMAGE"] = "who-need-help:caddy-production-" ENVIRON["GIT_SHA_VALUE"]
replacement["PRIMARY_DOMAIN"] = ENVIRON["DOMAIN"]

View File

@ -66,7 +66,11 @@ target_dir=$(dirname -- "$target")
compose_project_name=${TEST_COMPOSE_PROJECT_NAME:-who_need_help_test}
public_edge_network=${TEST_PUBLIC_EDGE_NETWORK:-who_need_help_public_edge}
public_route_id=${TEST_PUBLIC_ROUTE_ID:-who_need_help_test}
public_upstream_name=${TEST_PUBLIC_UPSTREAM_NAME:-who-need-help-test}
public_upstream_port=${TEST_PUBLIC_UPSTREAM_PORT:-4000}
public_health_path=${TEST_PUBLIC_HEALTH_PATH:-/healthz/ready}
public_www_redirect=${TEST_PUBLIC_WWW_REDIRECT:-false}
http_bind_address=${TEST_HTTP_BIND_ADDRESS:-127.0.0.1}
http_port=${TEST_HTTP_PORT:-4011}
mailpit_bind_address=${TEST_MAILPIT_BIND_ADDRESS:-127.0.0.1}
@ -91,7 +95,11 @@ git_sha=$(git -C "$ROOT" rev-parse --short=12 HEAD)
require_single_line_env_value TEST_COMPOSE_PROJECT_NAME "$compose_project_name"
require_single_line_env_value TEST_PUBLIC_EDGE_NETWORK "$public_edge_network"
require_single_line_env_value TEST_PUBLIC_ROUTE_ID "$public_route_id"
require_single_line_env_value TEST_PUBLIC_UPSTREAM_NAME "$public_upstream_name"
require_single_line_env_value TEST_PUBLIC_UPSTREAM_PORT "$public_upstream_port"
require_single_line_env_value TEST_PUBLIC_HEALTH_PATH "$public_health_path"
require_single_line_env_value TEST_PUBLIC_WWW_REDIRECT "$public_www_redirect"
require_single_line_env_value TEST_HTTP_BIND_ADDRESS "$http_bind_address"
require_single_line_env_value TEST_HTTP_PORT "$http_port"
require_single_line_env_value TEST_MAILPIT_BIND_ADDRESS "$mailpit_bind_address"
@ -122,10 +130,27 @@ require_single_line_env_value TEST_SUPPORT_INBOX_ADDRESS "$support_inbox_address
echo "TEST_PUBLIC_EDGE_NETWORK contains unsupported characters." >&2
exit 1
}
[[ "$public_route_id" =~ ^[a-z0-9][a-z0-9_-]{0,62}$ ]] || {
echo "TEST_PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
}
[[ "$public_upstream_name" =~ ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ ]] || {
echo "TEST_PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2
exit 1
}
if ! [[ "$public_upstream_port" =~ ^[0-9]+$ ]] ||
((public_upstream_port < 1 || public_upstream_port > 65535)); then
echo "TEST_PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
[[ "$public_health_path" =~ ^/[A-Za-z0-9._~:@%/+,=-]*$ ]] || {
echo "TEST_PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
}
[[ "$public_www_redirect" == true || "$public_www_redirect" == false ]] || {
echo "TEST_PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
}
[[ -n "$codex_session_id" ]] || {
echo "TEST_CODEX_SESSION_ID is required for the Build Week feedback page." >&2
exit 1
@ -269,7 +294,11 @@ DOMAIN=$domain \
GIT_SHA_VALUE=$git_sha \
COMPOSE_PROJECT_NAME_VALUE=$compose_project_name \
PUBLIC_EDGE_NETWORK_VALUE=$public_edge_network \
PUBLIC_ROUTE_ID_VALUE=$public_route_id \
PUBLIC_UPSTREAM_NAME_VALUE=$public_upstream_name \
PUBLIC_UPSTREAM_PORT_VALUE=$public_upstream_port \
PUBLIC_HEALTH_PATH_VALUE=$public_health_path \
PUBLIC_WWW_REDIRECT_VALUE=$public_www_redirect \
HTTP_BIND_ADDRESS_VALUE=$http_bind_address \
HTTP_PORT_VALUE=$http_port \
MAILPIT_BIND_ADDRESS_VALUE=$mailpit_bind_address \
@ -312,7 +341,11 @@ CODEX_SESSION_ID_VALUE=$codex_session_id \
replacement["HTTP_PORT"] = ENVIRON["HTTP_PORT_VALUE"]
replacement["PUBLIC_EDGE_ENABLED"] = "true"
replacement["PUBLIC_EDGE_NETWORK"] = ENVIRON["PUBLIC_EDGE_NETWORK_VALUE"]
replacement["PUBLIC_ROUTE_ID"] = ENVIRON["PUBLIC_ROUTE_ID_VALUE"]
replacement["PUBLIC_UPSTREAM_NAME"] = ENVIRON["PUBLIC_UPSTREAM_NAME_VALUE"]
replacement["PUBLIC_UPSTREAM_PORT"] = ENVIRON["PUBLIC_UPSTREAM_PORT_VALUE"]
replacement["PUBLIC_HEALTH_PATH"] = ENVIRON["PUBLIC_HEALTH_PATH_VALUE"]
replacement["PUBLIC_WWW_REDIRECT"] = ENVIRON["PUBLIC_WWW_REDIRECT_VALUE"]
replacement["MAILPIT_BIND_ADDRESS"] = ENVIRON["MAILPIT_BIND_ADDRESS_VALUE"]
replacement["MAILPIT_PORT"] = ENVIRON["MAILPIT_PORT_VALUE"]
replacement["DOCKER_SOCKET_GID"] = ENVIRON["DOCKER_SOCKET_GID_VALUE"]

View File

@ -71,7 +71,6 @@ sed -i \
-e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
-e "s|^CADDY_IMAGE=.*|CADDY_IMAGE=who-need-help:caddy-production-${MOCK_TARGET_COMMIT%${MOCK_TARGET_COMMIT#????????????}}|" \
"$env_file"
EOF
@ -158,18 +157,6 @@ if [ "$1" = inspect ]; then
esac
exit 0
fi
if [ "$1" = compose ]; then
case " $* " in
*' build edge '*)
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
*' up -d --no-deps --no-build --wait edge '*)
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
esac
fi
printf 'Unexpected docker invocation: %s\n' "$*" >&2
exit 1
EOF
@ -222,6 +209,8 @@ run_release() {
run_release forward_only >"$run_dir/forward-success.out"
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -F 'compose:build migrate' "$fixture/mock-commands.log" >/dev/null
grep -F 'compose:stop app' "$fixture/mock-commands.log" >/dev/null
grep -F 'compose:run --rm --no-deps migrate' \
@ -232,6 +221,15 @@ grep -R -F 'migration_policy=forward_only' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null
grep -R -F 'status=success' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null
if grep -R -E '^CADDY_IMAGE=' \
"$fixture/output/releases" --include rollback-manifest.txt >/dev/null; then
echo "Application release manifest unexpectedly captured the shared edge image." >&2
exit 1
fi
if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Application release drill touched the shared edge." >&2
exit 1
fi
write_old_env
printf '%s\n' "$current_commit" >"$fixture/git-state"
@ -252,6 +250,8 @@ grep -F 'previous application will not be restarted' \
"$run_dir/forward-failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 1
grep -R -F 'status=forward-only-release-failed' \
@ -276,6 +276,8 @@ grep -F 'restoring the previous immutable image tags' \
"$run_dir/safe-failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${current_commit:0:12}" \
"$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 2

View File

@ -207,7 +207,6 @@ rollback_manifest="$release_dir/rollback-manifest.txt"
printf 'APP_IMAGE=%s\n' "$(read_value APP_IMAGE)"
printf 'SOCKET_PROXY_IMAGE=%s\n' "$(read_value SOCKET_PROXY_IMAGE)"
printf 'POSTGIS_IMAGE=%s\n' "$(read_value POSTGIS_IMAGE)"
printf 'CADDY_IMAGE=%s\n' "$(read_value CADDY_IMAGE)"
printf 'migration_policy=%s\n' "$migration_policy"
printf 'migration_details=%s\n' \
"$(awk -F= '$1 == "migration_versions" {print $2}' <<<"$migration_policy_output")"
@ -228,13 +227,11 @@ restore_image_revision() {
APP_IMAGE_VALUE=$(awk -F= '$1 == "APP_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
SOCKET_PROXY_IMAGE_VALUE=$(awk -F= '$1 == "SOCKET_PROXY_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
POSTGIS_IMAGE_VALUE=$(awk -F= '$1 == "POSTGIS_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
CADDY_IMAGE_VALUE=$(awk -F= '$1 == "CADDY_IMAGE" {print substr($0,index($0,"=")+1)}' "$rollback_manifest") \
awk '
BEGIN {
replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"]
replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"]
replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"]
replacement["CADDY_IMAGE"] = ENVIRON["CADDY_IMAGE_VALUE"]
}
{
separator = index($0, "=")
@ -267,14 +264,6 @@ rollback_runtime() {
"$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${runtime_services[@]}" || true
edge_project=$(read_value EDGE_COMPOSE_PROJECT_NAME)
docker compose \
--project-name "$edge_project" \
--project-directory "$root" \
--env-file "$env_file" \
--file "$root/compose.edge.yaml" \
up -d --no-deps --no-build --wait edge || true
curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null || true
{
@ -312,15 +301,6 @@ revision_changed=true
"$root/scripts/check-environment-readiness.sh" "$env_file" --require-release
"$root/scripts/compose.sh" "$env_file" build "${build_services[@]}"
edge_project=$(read_value EDGE_COMPOSE_PROJECT_NAME)
edge_compose=(
docker compose
--project-name "$edge_project"
--project-directory "$root"
--env-file "$env_file"
--file "$root/compose.edge.yaml"
)
"${edge_compose[@]}" build edge
if [[ "$migration_policy" == "forward_only" ]]; then
echo "Stopping the old application before the forward-only migration boundary."
@ -332,7 +312,6 @@ migration_started=true
"$root/scripts/check-database.sh" "$env_file"
"$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${runtime_services[@]}"
"${edge_compose[@]}" up -d --no-deps --no-build --wait edge
COMPOSE_PROJECT_NAME="$compose_project" \
"$root/scripts/verify-realtime-cluster.sh" compose

View File

@ -43,6 +43,19 @@ else
exit 2
fi
legacy_edge_paths=(
compose.edge.yaml
deploy/caddy/Caddyfile
)
if ! git -C "$ROOT" diff --quiet \
"$remote_commit" "$local_commit" -- "${legacy_edge_paths[@]}"; then
echo "The application release contains shared edge routing changes." >&2
echo "The legacy edge serves both production and the frozen test domain." >&2
echo "Move the approved route change through the independent server-edge workflow." >&2
exit 2
fi
echo "Shared edge routing files are unchanged; application release will not manage Caddy."
migration_policy_output=$(
"$ROOT/scripts/release-migration-policy.sh" "$remote_commit" "$local_commit"
)

View File

@ -13,6 +13,7 @@ manifest="$remote_root/output/releases/release-1/rollback-manifest.txt"
target_commit=2222222222222222222222222222222222222222
previous_commit=1111111111111111111111111111111111111111
confirmation="whoneedhelp.com:$target_commit:$previous_commit"
edge_image=who-need-help:caddy-production-frozen-edge
cleanup() {
trap - EXIT HUP INT TERM
@ -39,7 +40,7 @@ printf '%s\n' \
"APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}" \
"POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}" \
"CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}" \
"CADDY_IMAGE=$edge_image" \
>"$fixture/.env"
backup="$fixture/output/backups/production/pre-release.dump"
@ -56,7 +57,6 @@ printf '%s\n' \
"APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \
"SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${previous_commit:0:12}" \
"POSTGIS_IMAGE=who-need-help:postgis-production-${previous_commit:0:12}" \
"CADDY_IMAGE=who-need-help:caddy-production-${previous_commit:0:12}" \
'migration_policy=application_safe' \
"database_backup=$remote_root/output/backups/production/pre-release.dump" \
'status=started' \
@ -75,7 +75,15 @@ shift
case "$*" in
'config --quiet') exit 0 ;;
'ps -q app') printf 'app-1\n'; exit 0 ;;
up\ *) printf 'compose:%s\n' "$*" >>"$MOCK_COMMAND_LOG"; exit 0 ;;
up\ *)
printf 'compose:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
if [ "${MOCK_FAIL_APP_UP:-}" = once ] &&
[ ! -e "$MOCK_FAIL_APP_MARKER" ]; then
: >"$MOCK_FAIL_APP_MARKER"
exit 17
fi
exit 0
;;
esac
printf 'Unexpected compose invocation: %s\n' "$*" >&2
exit 1
@ -111,7 +119,6 @@ if [ "$1" = inspect ]; then
'{{.Config.Image}}')
case "$container" in
app-1) awk -F= '$1 == "APP_IMAGE" {print substr($0, index($0, "=") + 1)}' "$MOCK_ENV_FILE" ;;
edge-1) awk -F= '$1 == "CADDY_IMAGE" {print substr($0, index($0, "=") + 1)}' "$MOCK_ENV_FILE" ;;
*) exit 1 ;;
esac
;;
@ -119,20 +126,6 @@ if [ "$1" = inspect ]; then
esac
exit 0
fi
if [ "$1" = compose ]; then
case " $* " in
*' ps -q edge ') printf 'edge-1\n'; exit 0 ;;
*' up -d --no-deps --no-build --wait edge ')
if [ "${MOCK_FAIL_EDGE_UP:-}" = once ] &&
[ ! -e "$MOCK_FAIL_EDGE_MARKER" ]; then
: >"$MOCK_FAIL_EDGE_MARKER"
exit 17
fi
printf 'edge:%s\n' "$*" >>"$MOCK_COMMAND_LOG"
exit 0
;;
esac
fi
printf 'Unexpected docker invocation: %s\n' "$*" >&2
exit 1
EOF
@ -193,11 +186,14 @@ docker run --rm \
grep -Fx "APP_IMAGE=who-need-help:production-${previous_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${previous_commit:0:12}" \
grep -Fx "CADDY_IMAGE=$edge_image" \
"$fixture/.env" >/dev/null
grep -F 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log" >/dev/null
grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null
if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Application rollback drill touched the shared edge." >&2
exit 1
fi
if grep -E -- 'migrate|--build|(^|[[:space:]])db([[:space:]]|$)' \
"$fixture/mock-commands.log" >/dev/null; then
echo "Rollback drill touched migrations, builds, or the database service." >&2
@ -242,9 +238,9 @@ sed -i \
-e "s|^APP_IMAGE=.*|APP_IMAGE=who-need-help:production-${target_commit:0:12}|" \
-e "s|^SOCKET_PROXY_IMAGE=.*|SOCKET_PROXY_IMAGE=who-need-help:socket-proxy-production-${target_commit:0:12}|" \
-e "s|^POSTGIS_IMAGE=.*|POSTGIS_IMAGE=who-need-help:postgis-production-${target_commit:0:12}|" \
-e "s|^CADDY_IMAGE=.*|CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}|" \
"$fixture/.env"
: >"$fixture/mock-commands.log"
rm -f "$fixture/mock-app-failed-once"
set +e
docker run --rm \
@ -255,8 +251,8 @@ docker run --rm \
--cap-drop ALL \
--security-opt no-new-privileges \
--env "WNH_PRODUCTION_ROLLBACK_CONFIRM=$confirmation" \
--env MOCK_FAIL_EDGE_UP=once \
--env "MOCK_FAIL_EDGE_MARKER=$remote_root/mock-edge-failed-once" \
--env MOCK_FAIL_APP_UP=once \
--env "MOCK_FAIL_APP_MARKER=$remote_root/mock-app-failed-once" \
"${container_env[@]}" \
"${container_mounts[@]}" \
"$BASE_IMAGE" \
@ -274,10 +270,13 @@ grep -F 'Rollback failed; restoring the pre-rollback image selection.' \
"$run_dir/failure.out" >/dev/null
grep -Fx "APP_IMAGE=who-need-help:production-${target_commit:0:12}" \
"$fixture/.env" >/dev/null
grep -Fx "CADDY_IMAGE=who-need-help:caddy-production-${target_commit:0:12}" \
grep -Fx "CADDY_IMAGE=$edge_image" \
"$fixture/.env" >/dev/null
test "$(grep -Fc 'compose:up -d --no-deps --no-build --wait app' \
"$fixture/mock-commands.log")" = 2
grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null
if grep -F 'edge:' "$fixture/mock-commands.log" >/dev/null; then
echo "Rollback recovery touched the shared edge." >&2
exit 1
fi
echo "Isolated production application rollback plan/apply/failure-recovery drill passed."

View File

@ -86,7 +86,6 @@ database_mode=$(read_unique "$env_file" DATABASE_MODE)
app_topology=$(read_unique "$env_file" APP_TOPOLOGY)
phx_host=$(read_unique "$env_file" PHX_HOST)
public_origin=$(read_unique "$env_file" WNH_BASE_URL)
edge_project=$(read_unique "$env_file" EDGE_COMPOSE_PROJECT_NAME)
[[ "$deployment_environment" == production ]] || {
echo "DEPLOYMENT_ENV is not production." >&2
@ -153,29 +152,23 @@ git -C "$root" merge-base --is-ancestor "$previous_commit" "$target_commit" || {
previous_app_image=$(read_unique "$manifest" APP_IMAGE)
previous_socket_image=$(read_unique "$manifest" SOCKET_PROXY_IMAGE)
previous_postgis_image=$(read_unique "$manifest" POSTGIS_IMAGE)
previous_caddy_image=$(read_unique "$manifest" CADDY_IMAGE)
require_image "$previous_app_image" production- APP_IMAGE
require_image "$previous_socket_image" socket-proxy-production- SOCKET_PROXY_IMAGE
require_image "$previous_postgis_image" postgis-production- POSTGIS_IMAGE
require_image "$previous_caddy_image" caddy-production- CADDY_IMAGE
target_short=${target_commit:0:12}
current_app_image=$(read_unique "$env_file" APP_IMAGE)
current_socket_image=$(read_unique "$env_file" SOCKET_PROXY_IMAGE)
current_postgis_image=$(read_unique "$env_file" POSTGIS_IMAGE)
current_caddy_image=$(read_unique "$env_file" CADDY_IMAGE)
[[ "$current_app_image" == "who-need-help:production-$target_short" &&
"$current_socket_image" == "who-need-help:socket-proxy-production-$target_short" &&
"$current_postgis_image" == "who-need-help:postgis-production-$target_short" &&
"$current_caddy_image" == "who-need-help:caddy-production-$target_short" ]] || {
"$current_postgis_image" == "who-need-help:postgis-production-$target_short" ]] || {
echo "Current production image selection does not match the manifest target commit." >&2
exit 2
}
for image in "$previous_app_image" "$previous_caddy_image"; do
docker image inspect "$image" >/dev/null
done
docker image inspect "$previous_app_image" >/dev/null
backup=$(realpath --canonicalize-existing "$backup")
case "$backup" in
@ -223,34 +216,7 @@ check_application() {
done
}
edge_compose=(
docker compose
--project-name "$edge_project"
--project-directory "$root"
--env-file "$env_file"
--file "$root/compose.edge.yaml"
)
check_edge() {
local expected_image=$1 container state health image
mapfile -t containers < <("${edge_compose[@]}" ps -q edge)
[[ ${#containers[@]} -gt 0 ]] || {
echo "Production edge service is not running." >&2
return 1
}
for container in "${containers[@]}"; do
state=$(docker inspect --format '{{.State.Status}}' "$container")
health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$container")
image=$(docker inspect --format '{{.Config.Image}}' "$container")
[[ "$state" == running && "$health" == healthy && "$image" == "$expected_image" ]] || {
echo "Production edge does not match the expected healthy image." >&2
return 1
}
done
}
check_application "$current_app_image"
check_edge "$current_caddy_image"
curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null
@ -264,8 +230,8 @@ printf 'Database mode: %s (no restore or migration reversal)\n' "$database_mode"
printf 'Rollback manifest: %s\n' "$manifest"
printf 'Verified backup evidence: %s\n' "$backup"
printf 'Exact confirmation: %s\n' "$confirmation"
echo "Scope: update four image selectors in production .env; recreate only application and edge containers."
echo "Excluded: Git checkout, database, migrations, test deployment, public Git, and Devpost."
echo "Scope: update three application image selectors in production .env; recreate only application containers."
echo "Excluded: shared edge/Caddy, Git checkout, database, migrations, test deployment, public Git, and Devpost."
if [[ "$action" == plan ]]; then
echo "Read-only production application rollback scope check passed."
@ -278,20 +244,18 @@ if [[ "${WNH_PRODUCTION_ROLLBACK_CONFIRM:-}" != "$confirmation" ]]; then
fi
update_images() {
local app_image=$1 socket_image=$2 postgis_image=$3 caddy_image=$4 temporary
local app_image=$1 socket_image=$2 postgis_image=$3 temporary
temporary=$(mktemp "$root/.env.image-selection.XXXXXX")
chmod 600 "$temporary"
APP_IMAGE_VALUE=$app_image \
SOCKET_PROXY_IMAGE_VALUE=$socket_image \
POSTGIS_IMAGE_VALUE=$postgis_image \
CADDY_IMAGE_VALUE=$caddy_image \
awk '
BEGIN {
replacement["APP_IMAGE"] = ENVIRON["APP_IMAGE_VALUE"]
replacement["SOCKET_PROXY_IMAGE"] = ENVIRON["SOCKET_PROXY_IMAGE_VALUE"]
replacement["POSTGIS_IMAGE"] = ENVIRON["POSTGIS_IMAGE_VALUE"]
replacement["CADDY_IMAGE"] = ENVIRON["CADDY_IMAGE_VALUE"]
}
{
separator = index($0, "=")
@ -326,12 +290,9 @@ recover_current_runtime() {
update_images \
"$current_app_image" \
"$current_socket_image" \
"$current_postgis_image" \
"$current_caddy_image" || true
"$current_postgis_image" || true
"$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${app_services[@]}" || true
"${edge_compose[@]}" \
up -d --no-deps --no-build --wait edge || true
curl --fail --silent --show-error --max-time 15 \
"https://$expected_domain/healthz/ready" >/dev/null || true
fi
@ -342,17 +303,13 @@ trap recover_current_runtime EXIT HUP INT TERM
update_images \
"$previous_app_image" \
"$previous_socket_image" \
"$previous_postgis_image" \
"$previous_caddy_image"
"$previous_postgis_image"
runtime_changed=true
"$root/scripts/compose.sh" "$env_file" \
up -d --no-deps --no-build --wait "${app_services[@]}"
"${edge_compose[@]}" \
up -d --no-deps --no-build --wait edge
check_application "$previous_app_image"
check_edge "$previous_caddy_image"
curl --fail --silent --show-error --max-time 30 \
"https://$expected_domain/healthz/ready" >/dev/null
curl --fail --silent --show-error --max-time 30 \

View File

@ -57,7 +57,7 @@ case "$deployment_env" in
required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE'
;;
production)
required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE CADDY_IMAGE'
required_keys='APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE'
;;
*)
echo "DEPLOYMENT_ENV must be test or production." >&2
@ -81,9 +81,6 @@ DEPLOYMENT_ENV_VALUE=$deployment_env GIT_SHA_VALUE=$git_sha awk '
replacement["APP_IMAGE"] = "who-need-help:" prefix "-" sha
replacement["SOCKET_PROXY_IMAGE"] = "who-need-help:socket-proxy-" prefix "-" sha
replacement["POSTGIS_IMAGE"] = "who-need-help:postgis-" prefix "-" sha
if (prefix == "production") {
replacement["CADDY_IMAGE"] = "who-need-help:caddy-production-" sha
}
}
{
separator = index($0, "=")

View File

@ -79,7 +79,7 @@ require_different() {
}
for key in COMPOSE_PROJECT_NAME APP_IMAGE SOCKET_PROXY_IMAGE POSTGIS_IMAGE \
PHX_HOST WNH_BASE_URL PUBLIC_UPSTREAM_NAME DATABASE_URL EMAIL_FROM_ADDRESS \
PHX_HOST WNH_BASE_URL PUBLIC_ROUTE_ID PUBLIC_UPSTREAM_NAME DATABASE_URL EMAIL_FROM_ADDRESS \
SECRET_KEY_BASE HANDOVER_SECRET RELEASE_COOKIE METRICS_TOKEN; do
require_different "$key"
done
@ -133,14 +133,16 @@ production_edge_network=$(read_env "$production_env" PUBLIC_EDGE_NETWORK)
exit 1
}
test_upstream=$(read_env "$test_env" PUBLIC_UPSTREAM_NAME)
test_upstream_port=$(read_env "$test_env" PUBLIC_UPSTREAM_PORT)
production_edge_test_upstream=$(read_env "$production_env" TEST_UPSTREAM)
[[ "$production_edge_test_upstream" == "$test_upstream:4000" ]] || {
[[ "$production_edge_test_upstream" == "$test_upstream:$test_upstream_port" ]] || {
echo "Production edge TEST_UPSTREAM does not point to the test alias." >&2
exit 1
}
production_upstream=$(read_env "$production_env" PUBLIC_UPSTREAM_NAME)
production_upstream_port=$(read_env "$production_env" PUBLIC_UPSTREAM_PORT)
production_edge_primary_upstream=$(read_env "$production_env" PRIMARY_UPSTREAM)
[[ "$production_edge_primary_upstream" == "$production_upstream:4000" ]] || {
[[ "$production_edge_primary_upstream" == "$production_upstream:$production_upstream_port" ]] || {
echo "Production edge PRIMARY_UPSTREAM does not point to production." >&2
exit 1
}

View File

@ -109,7 +109,11 @@ debug_base_url=$(require_value WNH_DEBUG_BASE_URL)
http_bind_address=$(require_value HTTP_BIND_ADDRESS)
public_edge_enabled=$(require_value PUBLIC_EDGE_ENABLED)
public_edge_network=$(require_value PUBLIC_EDGE_NETWORK)
public_route_id=$(require_value PUBLIC_ROUTE_ID)
public_upstream_name=$(require_value PUBLIC_UPSTREAM_NAME)
public_upstream_port=$(require_value PUBLIC_UPSTREAM_PORT)
public_health_path=$(require_value PUBLIC_HEALTH_PATH)
public_www_redirect=$(require_value PUBLIC_WWW_REDIRECT)
trusted_proxy_ips=$(optional_value TRAEFIK_TRUSTED_IPS)
postgres_password=$(optional_value POSTGRES_PASSWORD)
postgres_db=$(optional_value POSTGRES_DB)
@ -203,6 +207,23 @@ if [[ "$public_edge_enabled" == true ]]; then
echo "PUBLIC_UPSTREAM_NAME must be a lowercase Docker DNS alias." >&2
exit 1
}
[[ "$public_route_id" =~ ^[a-z0-9][a-z0-9_-]{0,62}$ ]] || {
echo "PUBLIC_ROUTE_ID contains unsupported characters." >&2
exit 1
}
if ! [[ "$public_upstream_port" =~ ^[0-9]+$ ]] ||
((public_upstream_port < 1 || public_upstream_port > 65535)); then
echo "PUBLIC_UPSTREAM_PORT must be between 1 and 65535." >&2
exit 1
fi
[[ "$public_health_path" =~ ^/[A-Za-z0-9._~:@%/+,=-]*$ ]] || {
echo "PUBLIC_HEALTH_PATH must be one absolute path." >&2
exit 1
}
[[ "$public_www_redirect" == true || "$public_www_redirect" == false ]] || {
echo "PUBLIC_WWW_REDIRECT must be true or false." >&2
exit 1
}
fi
[[ "$phx_host" == "$expected_domain" ]] || {
@ -557,7 +578,7 @@ reject_marker CODEX_SESSION_ID "$codex_session_id"
echo "PRIMARY_DOMAIN does not match EXPECTED_DOMAIN." >&2
exit 1
}
[[ "$primary_upstream" == "$(require_value PUBLIC_UPSTREAM_NAME):4000" ]] || {
[[ "$primary_upstream" == "$public_upstream_name:$public_upstream_port" ]] || {
echo "PRIMARY_UPSTREAM does not target the production application alias." >&2
exit 1
}

View File

@ -136,6 +136,22 @@ require_value EMAIL_FROM_ADDRESS >/dev/null
echo "The test public upstream alias must be who-need-help-test." >&2
exit 1
}
[[ "$(require_value PUBLIC_ROUTE_ID)" == who_need_help_test ]] || {
echo "The test route ID must be who_need_help_test." >&2
exit 1
}
[[ "$(require_value PUBLIC_UPSTREAM_PORT)" == 4000 ]] || {
echo "The test public upstream port must be 4000." >&2
exit 1
}
[[ "$(require_value PUBLIC_HEALTH_PATH)" == /healthz/ready ]] || {
echo "The test public health path must be /healthz/ready." >&2
exit 1
}
[[ "$(require_value PUBLIC_WWW_REDIRECT)" == false ]] || {
echo "The test route must not claim the primary www hostname." >&2
exit 1
}
google_id=$(read_value GOOGLE_OAUTH_CLIENT_ID 2>/dev/null || true)
google_secret=$(read_value GOOGLE_OAUTH_CLIENT_SECRET 2>/dev/null || true)