test: rehearse staging database upgrades

README.md

@@ -92,6 +92,18 @@ backup_path=$(printf '%s\n' "$backup_output" | sed -n 's/^Backup: //p')
 ./scripts/restore-drill-compose.sh "$backup_path"
 ```
 
+Rehearse the complete current release against that exact backup without
+changing the source Compose project:
+
+```bash
+./scripts/upgrade-rehearsal-compose.sh "$backup_path"
+```
+
+The rehearsal uses a uniquely named Compose project and database, applies the
+current migrations, starts 2 web and 2 worker replicas, checks the configured
+public-origin proxy behavior and cross-node PubSub, compares application-table
+counts, and removes its containers, networks, volume, and one-run image.
+
 For the reproducible encrypted S3-compatible drill, first start the isolated
 load project and then run:
 

compose.upgrade-rehearsal.yaml

@@ -0,0 +1,12 @@
+services:
+  migrate:
+    image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
+    pull_policy: never
+
+  web:
+    image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
+    pull_policy: never
+
+  worker:
+    image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
+    pull_policy: never

docs/local-hardening-plan.md

@@ -119,3 +119,11 @@ The goal remains open while any row lacks reproducible local evidence.
   paths, and the external protocol boundary. Scoped fixtures and one-run
   projects were removed, retained evidence is ignored and non-secret, and Git
   was clean at the tested application commit before this documentation update.
+- The public-facing database backup also passes a full isolated upgrade
+  rehearsal against application commit `af9018f`: the restored copy advanced
+  from 8 to all 10 migrations, exposed all 11 valid cursor indexes, ran two web
+  and two worker replicas in a four-node cluster, passed cross-node PubSub,
+  production HTTPS redirects, trusted-proxy pages, and both health endpoints,
+  and retained an empty application-table count diff. The scoped project,
+  volume, networks, and one-run image were removed; the ordinary Compose
+  project stayed on its original image and migration level.

docs/operations.md

@@ -153,6 +153,40 @@ clean operation, or the drill migration runner. The drill intentionally does
 not compare an older backup's row counts to the live source, because concurrent
 legitimate writes or a historical archive would make that comparison invalid.
 
+## Isolated Compose upgrade rehearsal
+
+After creating a current custom-format backup, run the complete current release
+against an isolated restored copy:
+
+```bash
+./scripts/upgrade-rehearsal-compose.sh \
+  output/backups/compose-YYYYMMDD-HHMMSS.dump
+```
+
+The rehearsal validates the checksum and archive catalog, reads the public
+origin configuration from ignored `.env`, and uses only the independently
+generated credentials in ignored mode-`0600` `.env.e2e`. It builds a uniquely
+tagged production release, creates a uniquely named Compose project and
+database from `template0`, restores the archive, records application-table
+counts, and then:
+
+1. applies every current timestamped Ecto migration;
+2. requires the localized category column and all 11 valid cursor indexes;
+3. starts 2 web and 2 worker replicas behind the isolated Traefik instance;
+4. requires the four-node BEAM cluster and cross-node PubSub probe;
+5. checks the production HTTP-to-HTTPS redirect, trusted-proxy public pages,
+   and both health endpoints;
+6. requires an empty before/after diff for every public application table
+   except the expected migration and Oban-internal tables;
+7. removes and verifies removal of the exact containers, networks, database
+   volume, and one-run image.
+
+The input archive is read-only and is not copied into the evidence directory.
+The ordinary Compose project, source database, public route, and running
+containers are outside the generated project scope. Non-secret evidence is
+retained under ignored mode-`0700`
+`output/upgrade-rehearsal/<run-id>/`, with files mode `0600`.
+
 ## Service checks
 
 ```bash

docs/verification.md

@@ -33,7 +33,7 @@ results from product limits and unknown production properties.
 - `mix compile --force --warnings-as-errors` and
   `mix format --check-formatted`: passed against the same final source.
 - `./scripts/quality.sh` passed ShellCheck 0.11.0, Hadolint 2.14.0 at warning
-  threshold, actionlint 1.7.12, all seven Compose renders, Helm lint, Trivy
+  threshold, actionlint 1.7.12, all eight Compose renders, Helm lint, Trivy
   source/rendered-manifest scanning, xref, Credo high-priority checks, Sobelow
   strict/private checks, Hex audit, 167 Phoenix tests, both npm audits, and the
   backup/MinIO/mc/external-mock/release-image vulnerability scans. The rendered
@@ -292,6 +292,19 @@ readiness, and removed that database. A final read-only source check still
 reported 8 migrations and the same user/request/message counts; the four
 running application container IDs were unchanged.
 
+The same archive then passed
+`./scripts/upgrade-rehearsal-compose.sh` against application commit `af9018f`.
+The isolated restored copy advanced from 8 to all 10 current migrations and
+reported all 11 cursor indexes valid. Two web and two worker replicas formed a
+four-node BEAM cluster, the cross-node PubSub probe passed, public requests
+produced the expected production HTTPS redirects, trusted-proxy pages and both
+health endpoints returned HTTP 200, and the application-table count diff was
+empty. The exact project, database volume, networks, and one-run image were
+absent after cleanup. Evidence is retained at
+`output/upgrade-rehearsal/20260719062507-3274364`. The ordinary Compose
+container IDs, 8-migration state, and user/request/message counts remained
+unchanged.
+
 After the Activity rollout, headed Chrome rendered the updated public
 navigation, followed the Activity link to the authenticated route, and received
 the expected login redirect and flash with zero console errors or warnings.
@@ -441,8 +454,10 @@ The complete local regression was repeated against application commit
 - The ordinary public-facing Compose project was checked read-only: local
   live/readiness, four-node PubSub, public HTTPS home/Safety/readiness, DNS, and
   TLS passed. Its current database was backed up and restore-tested through all
-  10 source migrations in a temporary database; its own existing rows,
+  10 source migrations, then ran the current 2-web/2-worker release on an
-  migration level, and running application containers remained unchanged.
+  isolated restored copy with an empty application-table diff and passing
+  HTTP/cluster/PubSub checks. Its own existing rows, migration level, and
+  running application containers remained unchanged.
 
 ## Known work before a public production launch
 

scripts/quality.sh

@@ -72,6 +72,9 @@ docker compose --env-file .env.example -f compose.yaml config --quiet
 mkdir -p "$scan_dir/e2e-output"
 E2E_OUTPUT_DIR="$scan_dir/e2e-output" docker compose --env-file .env.e2e.example \
   -f compose.yaml -f compose.e2e.yaml config --quiet
+REHEARSAL_IMAGE=who-need-help:rehearsal-render \
+  docker compose --env-file .env.e2e.example \
+  -f compose.yaml -f compose.upgrade-rehearsal.yaml config --quiet
 docker compose --env-file .env.load.example \
   -f compose.yaml -f compose.load.yaml config --quiet
 mkdir -p "$scan_dir/observability-runtime/prometheus" \

scripts/upgrade-rehearsal-compose.sh

@@ -0,0 +1,439 @@
+#!/bin/sh
+set -eu
+umask 077
+
+ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)
+cd "$ROOT"
+
+if [ "$#" -ne 1 ]; then
+  echo "Usage: $0 PATH_TO_CUSTOM_FORMAT_DUMP" >&2
+  exit 1
+fi
+
+dump=$1
+checksum="$dump.sha256"
+
+if [ ! -f "$dump" ] || [ ! -s "$dump" ]; then
+  echo "Dump does not exist or is empty: $dump" >&2
+  exit 1
+fi
+
+if [ ! -f "$checksum" ]; then
+  echo "Checksum manifest is required: $checksum" >&2
+  exit 1
+fi
+
+"$ROOT/scripts/ensure-local-e2e-env.sh" >/dev/null
+
+PUBLIC_ENV_FILE="$ROOT/.env"
+ENV_FILE="$ROOT/.env.e2e"
+
+if [ ! -f "$PUBLIC_ENV_FILE" ]; then
+  echo "Missing $PUBLIC_ENV_FILE." >&2
+  exit 1
+fi
+
+rehearsal_phx_host=$(
+  sh -c '. "$1"; printf %s "${PHX_HOST:?PHX_HOST is missing from .env}"' \
+    sh "$PUBLIC_ENV_FILE"
+)
+rehearsal_phx_scheme=$(
+  sh -c '. "$1"; printf %s "${PHX_SCHEME:?PHX_SCHEME is missing from .env}"' \
+    sh "$PUBLIC_ENV_FILE"
+)
+rehearsal_phx_url_port=$(
+  sh -c '. "$1"; printf %s "${PHX_URL_PORT:?PHX_URL_PORT is missing from .env}"' \
+    sh "$PUBLIC_ENV_FILE"
+)
+rehearsal_map_tile_url=$(
+  sh -c \
+    '. "$1"; printf %s "${MAP_TILE_URL:-https://tile.openstreetmap.org/{z}/{x}/{y}.png}"' \
+    sh "$PUBLIC_ENV_FILE"
+)
+
+set -a
+# shellcheck source=/dev/null
+. "$ENV_FILE"
+set +a
+
+: "${POSTGRES_USER:?POSTGRES_USER is missing from .env.e2e}"
+: "${DATABASE_URL:?DATABASE_URL is missing from .env.e2e}"
+
+dump_dir=$(CDPATH='' cd -- "$(dirname -- "$dump")" && pwd)
+dump_name=$(basename -- "$dump")
+
+(
+  cd "$dump_dir"
+  sha256sum --check --status "$(basename -- "$checksum")"
+)
+
+run_id="$(date -u +%Y%m%d%H%M%S)-$$"
+compact_id=$(printf '%s' "$run_id" | tr -d '-')
+project="who_need_help_upgrade_rehearsal_$compact_id"
+rehearsal_db="wnh_upgrade_rehearsal_$compact_id"
+output_dir="$ROOT/output/upgrade-rehearsal/$run_id"
+
+case "$project:$rehearsal_db" in
+  who_need_help_upgrade_rehearsal_*:wnh_upgrade_rehearsal_*) ;;
+  *)
+    echo "Refusing unexpected rehearsal identifiers." >&2
+    exit 1
+    ;;
+esac
+
+mkdir -p "$output_dir"
+chmod 700 "$ROOT/output" "$ROOT/output/upgrade-rehearsal" "$output_dir"
+
+case "$DATABASE_URL" in
+  *\?*)
+    database_base=${DATABASE_URL%%\?*}
+    database_query="?${DATABASE_URL#*\?}"
+    ;;
+
+  *)
+    database_base=$DATABASE_URL
+    database_query=
+    ;;
+esac
+
+database_prefix=${database_base%/*}
+
+if [ "$database_prefix" = "$database_base" ]; then
+  echo "Could not derive the isolated rehearsal DATABASE_URL." >&2
+  exit 1
+fi
+
+export DATABASE_URL="$database_prefix/$rehearsal_db$database_query"
+export HTTP_PORT=0
+export MAILPIT_PORT=0
+export PHX_HOST="$rehearsal_phx_host"
+export PHX_SCHEME="$rehearsal_phx_scheme"
+export PHX_URL_PORT="$rehearsal_phx_url_port"
+export MAP_TILE_URL="$rehearsal_map_tile_url"
+export TRAEFIK_TRUSTED_IPS=127.0.0.1/32
+export TRAEFIK_PROJECT_CONSTRAINT="$project"
+export TRAEFIK_APP_NAME="wnh-upgrade-rehearsal-$compact_id"
+export TRAEFIK_DOCKER_NETWORK="${project}_internal"
+export REHEARSAL_IMAGE="who-need-help:upgrade-rehearsal-$run_id"
+
+compose() {
+  docker compose \
+    --project-name "$project" \
+    --env-file "$ENV_FILE" \
+    --file "$ROOT/compose.yaml" \
+    --file "$ROOT/compose.upgrade-rehearsal.yaml" \
+    "$@"
+}
+
+cleanup() {
+  status=$?
+  trap - EXIT HUP INT TERM
+
+  if [ "$status" -ne 0 ]; then
+    compose ps --all >"$output_dir/compose-ps.txt" 2>&1 || true
+  fi
+
+  if ! compose down --volumes --remove-orphans >/dev/null 2>&1; then
+    echo "Could not remove the isolated rehearsal Compose project." >&2
+    if [ "$status" -eq 0 ]; then
+      status=1
+    fi
+  fi
+
+  if ! docker image rm "$REHEARSAL_IMAGE" >/dev/null 2>&1; then
+    if docker image inspect "$REHEARSAL_IMAGE" >/dev/null 2>&1; then
+      echo "Could not remove the isolated rehearsal image." >&2
+      if [ "$status" -eq 0 ]; then
+        status=1
+      fi
+    fi
+  fi
+
+  remaining_containers=$(
+    docker ps --all --quiet \
+      --filter "label=com.docker.compose.project=$project"
+  )
+  remaining_volumes=$(
+    docker volume ls --quiet \
+      --filter "label=com.docker.compose.project=$project"
+  )
+  remaining_networks=$(
+    docker network ls --quiet \
+      --filter "label=com.docker.compose.project=$project"
+  )
+
+  if [ -n "$remaining_containers$remaining_volumes$remaining_networks" ]; then
+    echo "Isolated rehearsal resources remain after cleanup." >&2
+    status=1
+  fi
+
+  exit "$status"
+}
+trap cleanup EXIT HUP INT TERM
+
+database_snapshot() {
+  target=$1
+  tables=$(
+    compose exec -T db \
+      psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+      --tuples-only --no-align \
+      --command "
+        SELECT tablename
+        FROM pg_tables
+        WHERE schemaname = 'public'
+          AND tablename NOT IN ('oban_jobs', 'schema_migrations', 'spatial_ref_sys')
+        ORDER BY tablename;
+      "
+  )
+
+  : >"$target"
+
+  for table in $tables; do
+    row_count=$(
+      compose exec -T db \
+        psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+        --tuples-only --no-align \
+        --command "SELECT count(*) FROM \"$table\";"
+    )
+    printf '%s %s\n' "$table" "$row_count" >>"$target"
+  done
+}
+
+compose config --quiet
+compose build migrate >"$output_dir/image-build.txt"
+compose up --detach --wait db
+
+compose exec -T db \
+  createdb \
+  --username "$POSTGRES_USER" \
+  --template template0 \
+  "$rehearsal_db"
+
+compose exec -T db \
+  pg_restore \
+  --username "$POSTGRES_USER" \
+  --dbname "$rehearsal_db" \
+  --exit-on-error \
+  --no-owner \
+  --no-privileges <"$dump"
+
+compose exec -T db pg_restore --list <"$dump" >"$output_dir/pg-restore-list.txt"
+
+database_snapshot "$output_dir/database-before.txt"
+compose exec -T db \
+  psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+  --tuples-only --no-align \
+  --command "SELECT version FROM schema_migrations ORDER BY version;" \
+  >"$output_dir/migrations-before.txt"
+
+compose up --detach --wait mailpit migrate proxy web worker
+
+database_snapshot "$output_dir/database-after.txt"
+diff -u "$output_dir/database-before.txt" "$output_dir/database-after.txt" \
+  >"$output_dir/database-diff.txt"
+
+expected_migrations=$(
+  find "$ROOT/priv/repo/migrations" -type f -name '[0-9]*.exs' \
+    -exec basename {} \; |
+    sed 's/_.*//' |
+    sort >"$output_dir/migrations-expected.txt"
+  wc -l <"$output_dir/migrations-expected.txt" | tr -d ' '
+)
+actual_migrations=$(
+  compose exec -T db \
+    psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+    --tuples-only --no-align \
+    --command "SELECT count(*) FROM schema_migrations;"
+)
+compose exec -T db \
+  psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+  --tuples-only --no-align \
+  --command "SELECT version FROM schema_migrations ORDER BY version;" \
+  >"$output_dir/migrations-after.txt"
+diff -u "$output_dir/migrations-expected.txt" "$output_dir/migrations-after.txt" \
+  >"$output_dir/migrations-diff.txt"
+
+description_columns=$(
+  compose exec -T db \
+    psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+    --tuples-only --no-align \
+    --command "
+      SELECT count(*)
+      FROM information_schema.columns
+      WHERE table_schema = 'public'
+        AND table_name = 'categories'
+        AND column_name = 'descriptions'
+        AND is_nullable = 'NO';
+    "
+)
+valid_cursor_indexes=$(
+  compose exec -T db \
+    psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
+    --tuples-only --no-align \
+    --command "
+      SELECT count(*)
+      FROM pg_class index_relation
+      JOIN pg_index index_metadata
+        ON index_metadata.indexrelid = index_relation.oid
+      JOIN pg_namespace namespace
+        ON namespace.oid = index_relation.relnamespace
+      WHERE namespace.nspname = 'public'
+        AND index_relation.relname IN (
+          'help_requests_discovery_cursor_index',
+          'help_requests_requester_cursor_index',
+          'activities_discovery_cursor_index',
+          'messages_assignment_cursor_index',
+          'activity_messages_activity_cursor_index',
+          'reviews_visible_cursor_index',
+          'reports_cursor_index',
+          'blocks_blocker_cursor_index',
+          'abuse_signals_status_cursor_index',
+          'category_proposals_cursor_index',
+          'users_moderation_cursor_index'
+        )
+        AND index_metadata.indisvalid
+        AND index_metadata.indisready;
+    "
+)
+
+if [ "$actual_migrations" != "$expected_migrations" ]; then
+  echo "Rehearsal migration count is $actual_migrations; expected $expected_migrations." >&2
+  exit 1
+fi
+
+if [ "$description_columns" != "1" ] || [ "$valid_cursor_indexes" != "11" ]; then
+  echo "The migrated rehearsal schema is incomplete." >&2
+  exit 1
+fi
+
+proxy_port=$(compose port proxy 80 | sed 's/.*://')
+
+case "$proxy_port" in
+  '' | *[!0-9]*)
+    echo "Could not determine the isolated proxy port." >&2
+    exit 1
+    ;;
+esac
+
+web_container=$(compose ps --quiet web | head -n 1)
+
+if [ -z "$web_container" ]; then
+  echo "No rehearsal web replica was found." >&2
+  exit 1
+fi
+
+case "$PHX_SCHEME:$PHX_URL_PORT" in
+  http:80 | https:443)
+    public_origin="$PHX_SCHEME://$PHX_HOST"
+    ;;
+
+  *)
+    public_origin="$PHX_SCHEME://$PHX_HOST:$PHX_URL_PORT"
+    ;;
+esac
+
+{
+  for path in / /safety; do
+    header_file="$output_dir/redirect-$(printf '%s' "$path" | tr '/' '_').txt"
+    status_code=$(
+      curl \
+        --silent \
+        --show-error \
+        --dump-header "$header_file" \
+        --output /dev/null \
+        --write-out '%{http_code}' \
+        --header "Host: $PHX_HOST" \
+        "http://127.0.0.1:$proxy_port$path"
+    )
+
+    location=$(
+      awk 'tolower($1) == "location:" {sub(/\r$/, "", $2); print $2}' \
+        "$header_file"
+    )
+
+    if [ "$status_code" != "301" ] || [ "$location" != "$public_origin$path" ]; then
+      echo "Unexpected production redirect for $path: HTTP $status_code $location." >&2
+      exit 1
+    fi
+
+    printf 'proxy %s %s %s\n' "$path" "$status_code" "$location"
+  done
+
+  for path in /healthz/live /healthz/ready; do
+    status_code=$(
+      curl \
+        --silent \
+        --show-error \
+        --output /dev/null \
+        --write-out '%{http_code}' \
+        --header "Host: $PHX_HOST" \
+        "http://127.0.0.1:$proxy_port$path"
+    )
+
+    if [ "$status_code" != "200" ]; then
+      echo "Unexpected HTTP $status_code for $path." >&2
+      exit 1
+    fi
+
+    printf 'proxy %s %s\n' "$path" "$status_code"
+  done
+
+  for path in / /safety; do
+    status_code=$(
+      docker exec "$web_container" \
+        curl \
+        --silent \
+        --show-error \
+        --output /dev/null \
+        --write-out '%{http_code}' \
+        --header "Host: $PHX_HOST" \
+        --header "X-Forwarded-Proto: $PHX_SCHEME" \
+        --header "X-Forwarded-Port: $PHX_URL_PORT" \
+        "http://127.0.0.1:4000$path"
+    )
+
+    if [ "$status_code" != "200" ]; then
+      echo "Unexpected trusted-proxy HTTP $status_code for $path." >&2
+      exit 1
+    fi
+
+    printf 'trusted-proxy %s %s\n' "$path" "$status_code"
+  done
+} >"$output_dir/http.txt"
+
+docker exec "$web_container" /app/bin/who_need_help rpc '
+  nodes = Enum.sort([node() | Node.list()])
+  IO.inspect(%{nodes: nodes, count: length(nodes)})
+  if length(nodes) != 4, do: exit({:unexpected_cluster_size, nodes})
+' >"$output_dir/cluster.txt"
+
+COMPOSE_PROJECT_NAME="$project" \
+  "$ROOT/scripts/verify-realtime-cluster.sh" compose \
+  >"$output_dir/pubsub.txt"
+
+image_id=$(docker image inspect --format '{{.Id}}' "$REHEARSAL_IMAGE")
+source_commit=$(git rev-parse HEAD)
+
+cat >"$output_dir/summary.json" <<EOF
+{
+  "status": "passed",
+  "source_commit": "$source_commit",
+  "image_id": "$image_id",
+  "input_dump": "$dump_name",
+  "expected_migrations": $expected_migrations,
+  "actual_migrations": $actual_migrations,
+  "migration_version_diff_bytes": 0,
+  "valid_cursor_indexes": $valid_cursor_indexes,
+  "application_table_diff_bytes": 0,
+  "web_replicas": 2,
+  "worker_replicas": 2,
+  "cluster_nodes": 4,
+  "pubsub": "passed",
+  "http": "passed"
+}
+EOF
+
+chmod 600 "$output_dir"/*
+
+echo "Isolated Compose upgrade rehearsal passed."
+echo "Evidence: $output_dir"