simpletest/who_need_help
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: rehearse staging database upgrades

Browse Source
This commit is contained in:
SimpleTest 2026-07-19 09:27:16 +03:00
parent af9018fc43
commit 143e159b01
7 changed files with 526 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@ -92,6 +92,18 @@ backup_path=$(printf '%s\n' "$backup_output" | sed -n 's/^Backup: //p')
./scripts/restore-drill-compose.sh "$backup_path"
```
Rehearse the complete current release against that exact backup without
changing the source Compose project:
```bash
./scripts/upgrade-rehearsal-compose.sh "$backup_path"
```
The rehearsal uses a uniquely named Compose project and database, applies the
current migrations, starts 2 web and 2 worker replicas, checks the configured
public-origin proxy behavior and cross-node PubSub, compares application-table
counts, and removes its containers, networks, volume, and one-run image.
For the reproducible encrypted S3-compatible drill, first start the isolated
load project and then run:

12
compose.upgrade-rehearsal.yaml Normal file
View File

@ -0,0 +1,12 @@
services:
migrate:
image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
pull_policy: never
web:
image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
pull_policy: never
worker:
image: ${REHEARSAL_IMAGE:?Set a unique REHEARSAL_IMAGE}
pull_policy: never

8
docs/local-hardening-plan.md
View File

@ -119,3 +119,11 @@ The goal remains open while any row lacks reproducible local evidence.
paths, and the external protocol boundary. Scoped fixtures and one-run
projects were removed, retained evidence is ignored and non-secret, and Git
was clean at the tested application commit before this documentation update.
- The public-facing database backup also passes a full isolated upgrade
rehearsal against application commit `af9018f`: the restored copy advanced
from 8 to all 10 migrations, exposed all 11 valid cursor indexes, ran two web
and two worker replicas in a four-node cluster, passed cross-node PubSub,
production HTTPS redirects, trusted-proxy pages, and both health endpoints,
and retained an empty application-table count diff. The scoped project,
volume, networks, and one-run image were removed; the ordinary Compose
project stayed on its original image and migration level.

34
docs/operations.md
View File

@ -153,6 +153,40 @@ clean operation, or the drill migration runner. The drill intentionally does
not compare an older backup's row counts to the live source, because concurrent
legitimate writes or a historical archive would make that comparison invalid.
## Isolated Compose upgrade rehearsal
After creating a current custom-format backup, run the complete current release
against an isolated restored copy:
```bash
./scripts/upgrade-rehearsal-compose.sh \
output/backups/compose-YYYYMMDD-HHMMSS.dump
```
The rehearsal validates the checksum and archive catalog, reads the public
origin configuration from ignored `.env`, and uses only the independently
generated credentials in ignored mode-`0600` `.env.e2e`. It builds a uniquely
tagged production release, creates a uniquely named Compose project and
database from `template0`, restores the archive, records application-table
counts, and then:
1. applies every current timestamped Ecto migration;
2. requires the localized category column and all 11 valid cursor indexes;
3. starts 2 web and 2 worker replicas behind the isolated Traefik instance;
4. requires the four-node BEAM cluster and cross-node PubSub probe;
5. checks the production HTTP-to-HTTPS redirect, trusted-proxy public pages,
and both health endpoints;
6. requires an empty before/after diff for every public application table
except the expected migration and Oban-internal tables;
7. removes and verifies removal of the exact containers, networks, database
volume, and one-run image.
The input archive is read-only and is not copied into the evidence directory.
The ordinary Compose project, source database, public route, and running
containers are outside the generated project scope. Non-secret evidence is
retained under ignored mode-`0700`
`output/upgrade-rehearsal/<run-id>/`, with files mode `0600`.
## Service checks
```bash

21
docs/verification.md
View File

@ -33,7 +33,7 @@ results from product limits and unknown production properties.
- `mix compile --force --warnings-as-errors` and
`mix format --check-formatted`: passed against the same final source.
- `./scripts/quality.sh` passed ShellCheck 0.11.0, Hadolint 2.14.0 at warning
threshold, actionlint 1.7.12, all seven Compose renders, Helm lint, Trivy
threshold, actionlint 1.7.12, all eight Compose renders, Helm lint, Trivy
source/rendered-manifest scanning, xref, Credo high-priority checks, Sobelow
strict/private checks, Hex audit, 167 Phoenix tests, both npm audits, and the
backup/MinIO/mc/external-mock/release-image vulnerability scans. The rendered
@ -292,6 +292,19 @@ readiness, and removed that database. A final read-only source check still
reported 8 migrations and the same user/request/message counts; the four
running application container IDs were unchanged.
The same archive then passed
`./scripts/upgrade-rehearsal-compose.sh` against application commit `af9018f`.
The isolated restored copy advanced from 8 to all 10 current migrations and
reported all 11 cursor indexes valid. Two web and two worker replicas formed a
four-node BEAM cluster, the cross-node PubSub probe passed, public requests
produced the expected production HTTPS redirects, trusted-proxy pages and both
health endpoints returned HTTP 200, and the application-table count diff was
empty. The exact project, database volume, networks, and one-run image were
absent after cleanup. Evidence is retained at
`output/upgrade-rehearsal/20260719062507-3274364`. The ordinary Compose
container IDs, 8-migration state, and user/request/message counts remained
unchanged.
After the Activity rollout, headed Chrome rendered the updated public
navigation, followed the Activity link to the authenticated route, and received
the expected login redirect and flash with zero console errors or warnings.
@ -441,8 +454,10 @@ The complete local regression was repeated against application commit
- The ordinary public-facing Compose project was checked read-only: local
live/readiness, four-node PubSub, public HTTPS home/Safety/readiness, DNS, and
TLS passed. Its current database was backed up and restore-tested through all
10 source migrations in a temporary database; its own existing rows,
migration level, and running application containers remained unchanged.
10 source migrations, then ran the current 2-web/2-worker release on an
isolated restored copy with an empty application-table diff and passing
HTTP/cluster/PubSub checks. Its own existing rows, migration level, and
running application containers remained unchanged.
## Known work before a public production launch

3
scripts/quality.sh
View File

@ -72,6 +72,9 @@ docker compose --env-file .env.example -f compose.yaml config --quiet
mkdir -p "$scan_dir/e2e-output"
E2E_OUTPUT_DIR="$scan_dir/e2e-output" docker compose --env-file .env.e2e.example \
-f compose.yaml -f compose.e2e.yaml config --quiet
REHEARSAL_IMAGE=who-need-help:rehearsal-render \
docker compose --env-file .env.e2e.example \
-f compose.yaml -f compose.upgrade-rehearsal.yaml config --quiet
docker compose --env-file .env.load.example \
-f compose.yaml -f compose.load.yaml config --quiet
mkdir -p "$scan_dir/observability-runtime/prometheus" \

439
scripts/upgrade-rehearsal-compose.sh Executable file
View File

@ -0,0 +1,439 @@
#!/bin/sh
set -eu
umask 077
ROOT=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)
cd "$ROOT"
if [ "$#" -ne 1 ]; then
echo "Usage: $0 PATH_TO_CUSTOM_FORMAT_DUMP" >&2
exit 1
fi
dump=$1
checksum="$dump.sha256"
if [ ! -f "$dump" ] || [ ! -s "$dump" ]; then
echo "Dump does not exist or is empty: $dump" >&2
exit 1
fi
if [ ! -f "$checksum" ]; then
echo "Checksum manifest is required: $checksum" >&2
exit 1
fi
"$ROOT/scripts/ensure-local-e2e-env.sh" >/dev/null
PUBLIC_ENV_FILE="$ROOT/.env"
ENV_FILE="$ROOT/.env.e2e"
if [ ! -f "$PUBLIC_ENV_FILE" ]; then
echo "Missing $PUBLIC_ENV_FILE." >&2
exit 1
fi
rehearsal_phx_host=$(
sh -c '. "$1"; printf %s "${PHX_HOST:?PHX_HOST is missing from .env}"' \
sh "$PUBLIC_ENV_FILE"
)
rehearsal_phx_scheme=$(
sh -c '. "$1"; printf %s "${PHX_SCHEME:?PHX_SCHEME is missing from .env}"' \
sh "$PUBLIC_ENV_FILE"
)
rehearsal_phx_url_port=$(
sh -c '. "$1"; printf %s "${PHX_URL_PORT:?PHX_URL_PORT is missing from .env}"' \
sh "$PUBLIC_ENV_FILE"
)
rehearsal_map_tile_url=$(
sh -c \
'. "$1"; printf %s "${MAP_TILE_URL:-https://tile.openstreetmap.org/{z}/{x}/{y}.png}"' \
sh "$PUBLIC_ENV_FILE"
)
set -a
# shellcheck source=/dev/null
. "$ENV_FILE"
set +a
: "${POSTGRES_USER:?POSTGRES_USER is missing from .env.e2e}"
: "${DATABASE_URL:?DATABASE_URL is missing from .env.e2e}"
dump_dir=$(CDPATH='' cd -- "$(dirname -- "$dump")" && pwd)
dump_name=$(basename -- "$dump")
(
cd "$dump_dir"
sha256sum --check --status "$(basename -- "$checksum")"
)
run_id="$(date -u +%Y%m%d%H%M%S)-$$"
compact_id=$(printf '%s' "$run_id" | tr -d '-')
project="who_need_help_upgrade_rehearsal_$compact_id"
rehearsal_db="wnh_upgrade_rehearsal_$compact_id"
output_dir="$ROOT/output/upgrade-rehearsal/$run_id"
case "$project:$rehearsal_db" in
who_need_help_upgrade_rehearsal_*:wnh_upgrade_rehearsal_*) ;;
*)
echo "Refusing unexpected rehearsal identifiers." >&2
exit 1
;;
esac
mkdir -p "$output_dir"
chmod 700 "$ROOT/output" "$ROOT/output/upgrade-rehearsal" "$output_dir"
case "$DATABASE_URL" in
*\?*)
database_base=${DATABASE_URL%%\?*}
database_query="?${DATABASE_URL#*\?}"
;;
*)
database_base=$DATABASE_URL
database_query=
;;
esac
database_prefix=${database_base%/*}
if [ "$database_prefix" = "$database_base" ]; then
echo "Could not derive the isolated rehearsal DATABASE_URL." >&2
exit 1
fi
export DATABASE_URL="$database_prefix/$rehearsal_db$database_query"
export HTTP_PORT=0
export MAILPIT_PORT=0
export PHX_HOST="$rehearsal_phx_host"
export PHX_SCHEME="$rehearsal_phx_scheme"
export PHX_URL_PORT="$rehearsal_phx_url_port"
export MAP_TILE_URL="$rehearsal_map_tile_url"
export TRAEFIK_TRUSTED_IPS=127.0.0.1/32
export TRAEFIK_PROJECT_CONSTRAINT="$project"
export TRAEFIK_APP_NAME="wnh-upgrade-rehearsal-$compact_id"
export TRAEFIK_DOCKER_NETWORK="${project}_internal"
export REHEARSAL_IMAGE="who-need-help:upgrade-rehearsal-$run_id"
compose() {
docker compose \
--project-name "$project" \
--env-file "$ENV_FILE" \
--file "$ROOT/compose.yaml" \
--file "$ROOT/compose.upgrade-rehearsal.yaml" \
"$@"
}
cleanup() {
status=$?
trap - EXIT HUP INT TERM
if [ "$status" -ne 0 ]; then
compose ps --all >"$output_dir/compose-ps.txt" 2>&1 || true
fi
if ! compose down --volumes --remove-orphans >/dev/null 2>&1; then
echo "Could not remove the isolated rehearsal Compose project." >&2
if [ "$status" -eq 0 ]; then
status=1
fi
fi
if ! docker image rm "$REHEARSAL_IMAGE" >/dev/null 2>&1; then
if docker image inspect "$REHEARSAL_IMAGE" >/dev/null 2>&1; then
echo "Could not remove the isolated rehearsal image." >&2
if [ "$status" -eq 0 ]; then
status=1
fi
fi
fi
remaining_containers=$(
docker ps --all --quiet \
--filter "label=com.docker.compose.project=$project"
)
remaining_volumes=$(
docker volume ls --quiet \
--filter "label=com.docker.compose.project=$project"
)
remaining_networks=$(
docker network ls --quiet \
--filter "label=com.docker.compose.project=$project"
)
if [ -n "$remaining_containers$remaining_volumes$remaining_networks" ]; then
echo "Isolated rehearsal resources remain after cleanup." >&2
status=1
fi
exit "$status"
}
trap cleanup EXIT HUP INT TERM
database_snapshot() {
target=$1
tables=$(
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "
SELECT tablename
FROM pg_tables
WHERE schemaname = 'public'
AND tablename NOT IN ('oban_jobs', 'schema_migrations', 'spatial_ref_sys')
ORDER BY tablename;
"
)
: >"$target"
for table in $tables; do
row_count=$(
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "SELECT count(*) FROM \"$table\";"
)
printf '%s %s\n' "$table" "$row_count" >>"$target"
done
}
compose config --quiet
compose build migrate >"$output_dir/image-build.txt"
compose up --detach --wait db
compose exec -T db \
createdb \
--username "$POSTGRES_USER" \
--template template0 \
"$rehearsal_db"
compose exec -T db \
pg_restore \
--username "$POSTGRES_USER" \
--dbname "$rehearsal_db" \
--exit-on-error \
--no-owner \
--no-privileges <"$dump"
compose exec -T db pg_restore --list <"$dump" >"$output_dir/pg-restore-list.txt"
database_snapshot "$output_dir/database-before.txt"
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "SELECT version FROM schema_migrations ORDER BY version;" \
>"$output_dir/migrations-before.txt"
compose up --detach --wait mailpit migrate proxy web worker
database_snapshot "$output_dir/database-after.txt"
diff -u "$output_dir/database-before.txt" "$output_dir/database-after.txt" \
>"$output_dir/database-diff.txt"
expected_migrations=$(
find "$ROOT/priv/repo/migrations" -type f -name '[0-9]*.exs' \
-exec basename {} \; |
sed 's/_.*//' |
sort >"$output_dir/migrations-expected.txt"
wc -l <"$output_dir/migrations-expected.txt" | tr -d ' '
)
actual_migrations=$(
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "SELECT count(*) FROM schema_migrations;"
)
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "SELECT version FROM schema_migrations ORDER BY version;" \
>"$output_dir/migrations-after.txt"
diff -u "$output_dir/migrations-expected.txt" "$output_dir/migrations-after.txt" \
>"$output_dir/migrations-diff.txt"
description_columns=$(
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "
SELECT count(*)
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'categories'
AND column_name = 'descriptions'
AND is_nullable = 'NO';
"
)
valid_cursor_indexes=$(
compose exec -T db \
psql --username "$POSTGRES_USER" --dbname "$rehearsal_db" \
--tuples-only --no-align \
--command "
SELECT count(*)
FROM pg_class index_relation
JOIN pg_index index_metadata
ON index_metadata.indexrelid = index_relation.oid
JOIN pg_namespace namespace
ON namespace.oid = index_relation.relnamespace
WHERE namespace.nspname = 'public'
AND index_relation.relname IN (
'help_requests_discovery_cursor_index',
'help_requests_requester_cursor_index',
'activities_discovery_cursor_index',
'messages_assignment_cursor_index',
'activity_messages_activity_cursor_index',
'reviews_visible_cursor_index',
'reports_cursor_index',
'blocks_blocker_cursor_index',
'abuse_signals_status_cursor_index',
'category_proposals_cursor_index',
'users_moderation_cursor_index'
)
AND index_metadata.indisvalid
AND index_metadata.indisready;
"
)
if [ "$actual_migrations" != "$expected_migrations" ]; then
echo "Rehearsal migration count is $actual_migrations; expected $expected_migrations." >&2
exit 1
fi
if [ "$description_columns" != "1" ] || [ "$valid_cursor_indexes" != "11" ]; then
echo "The migrated rehearsal schema is incomplete." >&2
exit 1
fi
proxy_port=$(compose port proxy 80 | sed 's/.*://')
case "$proxy_port" in
'' | *[!0-9]*)
echo "Could not determine the isolated proxy port." >&2
exit 1
;;
esac
web_container=$(compose ps --quiet web | head -n 1)
if [ -z "$web_container" ]; then
echo "No rehearsal web replica was found." >&2
exit 1
fi
case "$PHX_SCHEME:$PHX_URL_PORT" in
http:80 | https:443)
public_origin="$PHX_SCHEME://$PHX_HOST"
;;
*)
public_origin="$PHX_SCHEME://$PHX_HOST:$PHX_URL_PORT"
;;
esac
{
for path in / /safety; do
header_file="$output_dir/redirect-$(printf '%s' "$path" | tr '/' '_').txt"
status_code=$(
curl \
--silent \
--show-error \
--dump-header "$header_file" \
--output /dev/null \
--write-out '%{http_code}' \
--header "Host: $PHX_HOST" \
"http://127.0.0.1:$proxy_port$path"
)
location=$(
awk 'tolower($1) == "location:" {sub(/\r$/, "", $2); print $2}' \
"$header_file"
)
if [ "$status_code" != "301" ] || [ "$location" != "$public_origin$path" ]; then
echo "Unexpected production redirect for $path: HTTP $status_code $location." >&2
exit 1
fi
printf 'proxy %s %s %s\n' "$path" "$status_code" "$location"
done
for path in /healthz/live /healthz/ready; do
status_code=$(
curl \
--silent \
--show-error \
--output /dev/null \
--write-out '%{http_code}' \
--header "Host: $PHX_HOST" \
"http://127.0.0.1:$proxy_port$path"
)
if [ "$status_code" != "200" ]; then
echo "Unexpected HTTP $status_code for $path." >&2
exit 1
fi
printf 'proxy %s %s\n' "$path" "$status_code"
done
for path in / /safety; do
status_code=$(
docker exec "$web_container" \
curl \
--silent \
--show-error \
--output /dev/null \
--write-out '%{http_code}' \
--header "Host: $PHX_HOST" \
--header "X-Forwarded-Proto: $PHX_SCHEME" \
--header "X-Forwarded-Port: $PHX_URL_PORT" \
"http://127.0.0.1:4000$path"
)
if [ "$status_code" != "200" ]; then
echo "Unexpected trusted-proxy HTTP $status_code for $path." >&2
exit 1
fi
printf 'trusted-proxy %s %s\n' "$path" "$status_code"
done
} >"$output_dir/http.txt"
docker exec "$web_container" /app/bin/who_need_help rpc '
nodes = Enum.sort([node() | Node.list()])
IO.inspect(%{nodes: nodes, count: length(nodes)})
if length(nodes) != 4, do: exit({:unexpected_cluster_size, nodes})
' >"$output_dir/cluster.txt"
COMPOSE_PROJECT_NAME="$project" \
"$ROOT/scripts/verify-realtime-cluster.sh" compose \
>"$output_dir/pubsub.txt"
image_id=$(docker image inspect --format '{{.Id}}' "$REHEARSAL_IMAGE")
source_commit=$(git rev-parse HEAD)
cat >"$output_dir/summary.json" <<EOF
{
"status": "passed",
"source_commit": "$source_commit",
"image_id": "$image_id",
"input_dump": "$dump_name",
"expected_migrations": $expected_migrations,
"actual_migrations": $actual_migrations,
"migration_version_diff_bytes": 0,
"valid_cursor_indexes": $valid_cursor_indexes,
"application_table_diff_bytes": 0,
"web_replicas": 2,
"worker_replicas": 2,
"cluster_nodes": 4,
"pubsub": "passed",
"http": "passed"
}
EOF
chmod 600 "$output_dir"/*
echo "Isolated Compose upgrade rehearsal passed."
echo "Evidence: $output_dir"