# Operations runbook This runbook describes the commands that are implemented and verified in this repository. It does not claim a production recovery point objective, recovery time objective, retention period, storage capacity, or high-availability model; those values require product policy and measurements from the eventual production environment. ## Private Gitea continuous integration The observed private Git remote is hosted by Gitea 1.22.0. Gitea 1.22 reads repository workflows from `.gitea/workflows/`, so the repository keeps a Gitea-specific quality workflow in addition to the GitHub-compatible workflow. Both files are checked by the pinned local actionlint gate. The Gitea workflow deliberately targets only the custom `who-need-help-ci` runner label. Its jobs build and run many Docker images and must not be scheduled onto an unrelated shared runner or onto the production application host. Its gates run sequentially in one job so a single push cannot start four heavy Docker suites concurrently. Before enabling it: 1. enable Actions in the private repository settings; 2. register a trusted runner controlled by this project; 3. assign `who-need-help-ci:host` only to a Linux host where the exact Docker Engine and Compose commands used by the repository have been verified; 4. install the runtime required by the pinned checkout action on that host; 5. keep the runner registration state and any Docker socket access outside the repository and away from untrusted projects. The runner's availability, labels, software versions, free resources and repository Actions setting cannot be established from the unauthenticated repository clone. Do not treat the presence of the workflow file as evidence that remote CI has run. After provisioning the runner, trigger the workflow manually, require the full job and all four gate steps to pass, and record the run URL and commit before protecting `main`. ## First production Compose environment The deployment environment selects topology and database ownership: | Setting | Result | | --- | --- | | `APP_TOPOLOGY=compact` | One Phoenix+Oban container, directly published to the host; default for the first server. | | `APP_TOPOLOGY=split` | Traefik, `WEB_REPLICAS` web containers, and `WORKER_REPLICAS` worker containers. | | `DATABASE_MODE=container` | Start the project-owned PostgreSQL/PostGIS container and volume. | | `DATABASE_MODE=external` | Do not create/start a Compose database; use only `DATABASE_URL`. | There is no Redis dependency. Queues, rate-limit counters, Oban leadership, and durable application state use PostgreSQL. ### Development public route after a workstation restart The current development gateway sends `whoneedhelp.imalto.site` to the workstation's assigned OpenVPN address `10.8.0.14:4010`. If HTTPS connects to the gateway but returns no response after a workstation restart, first verify the local app and tunnel: ```bash curl --fail --show-error --max-time 10 \ http://127.0.0.1:4010/healthz/ready ip -brief address show tun0 nmcli --wait 30 connection up openvpn__toha_nobara_pc curl --fail --show-error --max-time 15 \ https://whoneedhelp.imalto.site/healthz/ready ``` The observed profile requires `remote-cert-tls=server`; do not remove that server-certificate check. NetworkManager documents that `connection.autoconnect` is not implemented for VPN profiles and recommends a base connection's `connection.secondaries` instead: [NetworkManager connection settings](https://www.networkmanager.dev/docs/api/latest/nm-settings-nmcli.html). The observed wired base profile currently has no secondary connection, so a standalone `connection.autoconnect=yes` value on the VPN does not prove that the development route will return after reboot. Adding the VPN UUID as a wired/Wi-Fi secondary is a workstation networking change: this profile currently installs the default route and DNS through the VPN. Review that whole-host effect and obtain explicit approval before configuring automatic activation. The manual command above changes no project, test, production, or gateway configuration. ### Two independent checkouts and one `.env` in each The server uses exactly these independent Git clones: ```text /srv/who_need_help-test/.env /srv/who_need_help-production/.env ``` No `.env.test`, `.env.staging`, or `.env.production` is used inside either application checkout. Each checkout can remain on a different commit. Test and production have different Compose projects, application/infrastructure image tags, database connections, Docker volumes, public aliases, Google OAuth clients, email delivery paths, and generated secrets. Oban queues are isolated by those different PostgreSQL databases. There is no Redis dependency. Generated harness state is not a deployment environment. E2E, load, and Android instrumentation scripts keep their random local inputs below the ignored mode-`0700` `output/runtime/` directory: ```text output/runtime/e2e.env output/runtime/load.env output/runtime/android-test.env ``` Those files are generated automatically, never copied to a server, and do not represent dev, test, or production. The one ignored `.env` at each checkout root remains the only application/deployment configuration. The current submitted deployment still has a legacy shared Caddy container created from the production checkout. Keep it running and unchanged while the judging test URL is frozen. Application release and rollback scripts do not build, recreate, or select an image for that container. The replacement is a separate server-level `server_edge` project with its own single mode-`0600` `.env`, route directory, Caddy image, certificate volumes, and Compose lifecycle. Each application keeps only its route contract in its existing `.env`: `PHX_HOST`, `PUBLIC_ROUTE_ID`, `PUBLIC_EDGE_NETWORK`, `PUBLIC_UPSTREAM_NAME`, `PUBLIC_UPSTREAM_PORT`, `PUBLIC_HEALTH_PATH`, and `PUBLIC_WWW_REDIRECT`. No application deployment is allowed to restart the server edge. Both applications intentionally share only the external public edge Docker network. ### Environment-specific Android builds and App Links Android build inputs belong in the same ignored mode-`0600` `.env` as the web checkout they target. Do not create `.env.android-release`, `.env.production`, or another permanent environment file: ```dotenv WNH_BASE_URL=https://dev.example.com WNH_ANDROID_VERSION_CODE=1 WNH_ANDROID_VERSION_NAME=0.1.0 WNH_ANDROID_DEVELOPMENT_SIGNING_KEY_ALIAS=who-need-help-development ANDROID_APP_LINKS_PACKAGE_NAME=org.whoneedhelp.mobile.development ANDROID_APP_LINKS_SHA256_CERT_FINGERPRINTS=AA:BB:... ``` The development checkout uses package `org.whoneedhelp.mobile.development` and a dedicated stable key under `~/.config/who_need_help/android-development/`. Generate it once and place only its public identity into the existing ignored `.env`: ```bash ./scripts/init-android-development-signing.sh ./scripts/configure-android-development-env.sh ./scripts/android-development-build.sh ``` The build exports the package and certificate reports, verifies that both match the checkout `.env`, and checks the HTTPS `/.well-known/assetlinks.json` response. Losing this key changes the development certificate and breaks previously installed App Links, so back it up. The independent test checkout uses package `org.whoneedhelp.mobile.staging`, its own key under `~/.config/who_need_help/android-staging/`, and `scripts/android-staging-build.sh`. Do not copy either ignored `.env` or signing material between development and test. The production checkout instead uses package `org.whoneedhelp.mobile`, the separate upload material under `~/.config/who_need_help/android-release/`, and its own `.env`: ```dotenv WNH_ANDROID_SIGNING_KEY_ALIAS=who-need-help-upload ANDROID_APP_LINKS_PACKAGE_NAME=org.whoneedhelp.mobile ANDROID_APP_LINKS_SHA256_CERT_FINGERPRINTS=UPLOAD_SHA256,PLAY_APP_SIGNING_SHA256 ANDROID_PLAY_APP_SIGNING_SHA256_CERT_FINGERPRINTS=PLAY_APP_SIGNING_SHA256 ``` Run `./scripts/android-release-build.sh` from that production release checkout. It produces an APK, Play AAB, package report, signing report, and lint report. Before the Play application exists, the build may use only the upload certificate in `ANDROID_APP_LINKS_SHA256_CERT_FINGERPRINTS` and leave `ANDROID_PLAY_APP_SIGNING_SHA256_CERT_FINGERPRINTS` empty. This is sufficient to create the first signed AAB, but `check-environment-readiness.sh --require-release` intentionally continues to report the Play identity as missing. Application-only server releases use `--require-server-release` plus the production validator's `--allow-pre-play` mode: they accept this upload-certificate-only state while continuing to reject every other missing production capability. These modes do not make an Android build ready for Google Play. After Play App Signing is enabled, add the Play signing certificate fingerprint to the comma-separated App Links value; the upload certificate alone does not describe Play-delivered APKs. Record the same Play fingerprint separately in `ANDROID_PLAY_APP_SIGNING_SHA256_CERT_FINGERPRINTS`; release readiness verifies that every Play identity is present in the public App Links list. The production environment validator accepts multiple SHA-256 fingerprints and rejects partial, malformed, upload-only, or inconsistent configuration. ### Release capability inputs Each environment owns distinct external-provider credentials. Seed a new production `.env` with `scripts/init-production-env.sh` and the corresponding `PRODUCTION_*` process variables; use `TEST_*` only when creating the separate test checkout. The generated file uses the ordinary runtime names: | Capability | Values kept in that checkout's `.env` | | --- | --- | | Google sign-in | `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET`, `GOOGLE_OAUTH_AUTHORIZED_PARTY_IDS` | | Browser push | three `WEB_PUSH_VAPID_*` values | | Android Firebase client | four public `WNH_FIREBASE_*` values | | Android delivery | `FCM_PROJECT_ID` and one private service-account source | | Verified Android links | `ANDROID_APP_LINKS_PACKAGE_NAME`, `ANDROID_APP_LINKS_SHA256_CERT_FINGERPRINTS` | | Transactional email | `SMTP_*`, sender, and `SUPPORT_INBOX_ADDRESS` | Do not reuse a Google client, VAPID private key, Firebase project/service account, SMTP credential, or Android signing key between dev and production. The public Firebase Android values are build configuration; the Base64 FCM service-account JSON is a server secret and must never be passed into the Android build. For Android Google sign-in, each environment's `GOOGLE_OAUTH_CLIENT_ID` is also the public server client ID supplied at runtime to Credential Manager. `GOOGLE_OAUTH_CLIENT_SECRET` never leaves Phoenix. The Android package and signing-certificate identity must be registered in the matching Google project: `org.whoneedhelp.mobile.development` plus the stable development certificate for development, `org.whoneedhelp.mobile.staging` plus the stable staging certificate for test, and `org.whoneedhelp.mobile` plus the Play-distributed certificate for production. Do not add a second Google secret file or Gradle property. Set `GOOGLE_OAUTH_AUTHORIZED_PARTY_IDS` to the comma-separated Android OAuth client IDs from that environment. They are accepted only as the signed `azp` claim; the signed `aud` must still contain the environment's Web client ID. Check an environment without printing its secret values: ```bash ./scripts/check-environment-readiness.sh .env ./scripts/check-environment-readiness.sh .env --require-release ``` The complete promotion sequence and the human/provider decisions that these scripts cannot prove are listed in [`docs/public-launch-checklist.md`](public-launch-checklist.md). Provider downloads can be imported into that same file without placing secrets on a command line or printing them: ```bash ./scripts/generate-vapid-env.sh \ .env mailto:contact@YOUR_DOMAIN chmod 600 /secure/downloads/google-oauth-client.json ./scripts/import-google-oauth-client.sh \ .env /secure/downloads/google-oauth-client.json ./scripts/import-firebase-android-config.sh \ .env /secure/downloads/google-services.json chmod 600 /secure/downloads/fcm-service-account.json ./scripts/import-fcm-service-account.sh \ .env /secure/downloads/fcm-service-account.json ``` The VAPID helper runs the exact locked `web_push_elixir` generator in an isolated, network-disabled container, imports the result atomically, removes its one-run image tag and temporary files, and never prints either key. It refuses to replace an existing VAPID identity because an unplanned rotation invalidates existing browser subscriptions. The importers validate the exact OAuth callback, Android package, Firebase project relationship, and required service-account fields before atomically replacing existing keys. They preserve mode `0600` and never create another permanent environment file. Provider and initializer values that cannot be represented as one unquoted Compose `.env` line are rejected before mutation. Literal dollar signs are stored as `$$`, which Compose resolves back to one `$` inside the container. OAuth and service-account downloads still contain private credentials after import; deliberately move them to protected backup storage or remove them after verification. The first command reports incomplete or local-only capabilities. The second is a blocking release preflight and exits nonzero until Google sign-in, external SMTP, browser Web Push, Android Firebase/FCM, App Links, support routing, Android version/signing aliases, and the core application configuration are all complete. Create the test configuration inside the test checkout: ```bash cd /srv/who_need_help-test TEST_CODEX_SESSION_ID=YOUR_MAIN_CODEX_SESSION_ID \ TEST_GOOGLE_OAUTH_CLIENT_ID=YOUR_TEST_CLIENT_ID \ TEST_GOOGLE_OAUTH_CLIENT_SECRET=YOUR_TEST_CLIENT_SECRET \ TEST_GOOGLE_OAUTH_AUTHORIZED_PARTY_IDS=YOUR_TEST_ANDROID_CLIENT_ID \ ./scripts/init-test-env.sh test.whoneedhelp.com ./scripts/validate-test-env.sh .env test.whoneedhelp.com ./scripts/deploy-up.sh .env ``` For later test updates, check out the desired clean revision and update only the image tags. Existing deployment secrets remain unchanged: ```bash ./scripts/set-deployment-revision.sh .env ./scripts/deploy-up.sh .env ``` Test always uses its own `who_need_help_test` PostGIS container/volume. Local development can use Mailpit; a public test deployment must use its own SMTP password and a visibly test-specific sender identity. Create the production configuration inside the production checkout after loading only the production database/provider credentials into the process: ```bash cd /srv/who_need_help-production PRODUCTION_DATABASE_MODE=external \ PRODUCTION_DATABASE_URL='ecto://PRODUCTION_ROLE:PASSWORD@localhost/who_need_help_production' \ PRODUCTION_DATABASE_SOCKET_DIR=/var/run/postgresql \ PRODUCTION_EMAIL_DELIVERY_PROVIDER=smtp \ PRODUCTION_SMTP_RELAY=smtp.example.net \ PRODUCTION_SMTP_PORT=587 \ PRODUCTION_SMTP_USERNAME=YOUR_PRODUCTION_SMTP_LOGIN \ PRODUCTION_SMTP_PASSWORD=YOUR_PRODUCTION_SMTP_PASSWORD \ PRODUCTION_GOOGLE_OAUTH_CLIENT_ID=YOUR_PRODUCTION_CLIENT_ID \ PRODUCTION_GOOGLE_OAUTH_CLIENT_SECRET=YOUR_PRODUCTION_CLIENT_SECRET \ PRODUCTION_GOOGLE_OAUTH_AUTHORIZED_PARTY_IDS=YOUR_PRODUCTION_ANDROID_CLIENT_ID \ PRODUCTION_CODEX_SESSION_ID=YOUR_MAIN_CODEX_SESSION_ID \ ./scripts/init-production-env.sh whoneedhelp.com ./scripts/validate-production-env.sh .env whoneedhelp.com ``` Promote the exact revision already verified in test by checking out that SHA in the independent production clone, then update only the production image tags: ```bash git checkout --detach ./scripts/set-deployment-revision.sh .env ./scripts/deploy-up.sh .env ``` Before either public switch, prove cross-environment isolation without printing credentials: ```bash /srv/who_need_help-test/scripts/validate-deployment-isolation.sh \ /srv/who_need_help-test /srv/who_need_help-production ``` Start/update test first and run the complete browser/API/Android verification. Only then check out that exact tested SHA in production. Start the application with `./scripts/deploy-up.sh .env`. The application command joins the external public network but does not own or restart Caddy. The legacy `./scripts/edge-up.sh .env` command exists only for the currently frozen submitted deployment. Do not use it from an ordinary application release. After judging, render and validate each route through the independent `server_edge` workflow, transfer certificate-volume ownership in a reviewed maintenance window, and only then retire the legacy edge container. The authoritative A records for `whoneedhelp.com`, `www.whoneedhelp.com`, and `test.whoneedhelp.com` must point to the verified server address before Caddy can obtain their certificates. Do not remove the edge volumes during an ordinary application deploy. Do not add `--volumes` to a stop command unless the exact database has been inspected and deletion is intended. The initializer reads `.env.example`, derives the numeric group of that host's Docker socket, generates independent random values for PostgreSQL, `SECRET_KEY_BASE`, handover codes, the BEAM release cookie, and metrics access, and writes an ignored mode-`0600` file. It does not print those values and refuses to overwrite an existing destination. By default, the generated public proxy port binds to `127.0.0.1`, which is appropriate only when the verified reverse proxy reaches the application on the same host. Set `PRODUCTION_HTTP_BIND_ADDRESS` when generating the file, or edit `HTTP_BIND_ADDRESS` afterward, to match the observed target topology. Replace `TRAEFIK_TRUSTED_IPS` with the exact source IP/CIDR observed at Traefik; do not copy the temporary VPN value into an unrelated server. Configure the transactional SMTP relay and a sender accepted by it using `EMAIL_DELIVERY_PROVIDER=smtp` plus the `SMTP_*` settings. The initializer accepts the corresponding `PRODUCTION_EMAIL_DELIVERY_PROVIDER` and `PRODUCTION_SMTP_*` inputs. Set optional `SUPPORT_INBOX_ADDRESS` to a monitored address for support/removal queue alerts and email `Reply-To`; leaving it empty disables operator email alerts, not the protected queues. If Google registration/sign-in is enabled, also set both Google Web client credentials and register `https://YOUR_PHX_HOST/auth/google/callback` as the exact authorized redirect URI. Leave both credentials empty to keep the feature disabled. Then run: ```bash ./scripts/validate-production-env.sh .env whoneedhelp.com ./scripts/deploy-up.sh .env ``` The production override keeps Mailpit behind its inactive `local-mail` profile, so public registration cannot appear to succeed while mail is only retained locally. The validator checks file ownership/mode, origin consistency, template markers, independent generated secrets, settings required by the selected email provider, a complete-or-empty Google credential pair, the generated database URL, and the final Compose render without printing secrets. It does not contact DNS, TLS, the email provider, the reverse proxy, or the application. After deployment, verify `/healthz/ready`, inspect all replica health and logs, register a unique address through the public browser, receive its message at the real mailbox, follow the HTTPS confirmation link, and remove only that run-scoped account. ### External PostgreSQL/PostGIS #### PostgreSQL on the same Linux host The server-local mode uses the PostgreSQL Unix socket instead of exposing the database on a Docker-reachable TCP address. Ecto/Postgrex receive `DATABASE_SOCKET_DIR`; `scripts/compose.sh` then adds `compose.external-db-socket.yaml` and mounts that exact directory read-only into the active application and migration services. An empty setting preserves the normal remote-provider TCP behavior. After read-only inspection confirms the intended PostgreSQL 18 cluster, installed PostGIS package, socket directory, and absence of the project-scoped roles/databases, run the root-only provisioner: ```bash sudo ./scripts/provision-host-postgres.sh "$USER" ``` Its exact mutation scope is: - prepend two database-and-role-specific `local ... scram-sha-256` rules to the active `pg_hba.conf`, retaining a mode-`0600` copy under `/var/backups/who_need_help/`, and reload that cluster; - create login roles `wnh_production` and `wnh_test` without superuser, database-creation, role-creation, replication, or row-security bypass rights; - create empty owner databases `who_need_help_production` and `who_need_help_test`, revoke public connect, and preload `citext` and PostGIS so the non-superuser Ecto migrations can run; - verify both credentials through the observed Unix socket and write separate mode-`0600` initializer fragments under `~/.config/who_need_help/` without printing passwords. The command refuses to overwrite credential fragments, refuses any matching pre-existing role/database or managed HBA marker, validates the candidate HBA rules before reload, and restores its HBA backup plus removes only objects it created if provisioning fails. It does not inspect or migrate application tables; the release migration runner remains authoritative for schema. Generate test or production after loading only the matching trusted fragment: ```bash set -a . "$HOME/.config/who_need_help/database-production.env" set +a ./scripts/init-production-env.sh whoneedhelp.com unset PRODUCTION_DATABASE_MODE PRODUCTION_DATABASE_URL \ PRODUCTION_DATABASE_SOCKET_DIR ``` For test external-database drills, load `database-test.env`; the public test deployment normally uses its own Compose database. Do not load both fragments into one shell. Provision the database and role first, then generate the environment without placing its credentials on a command line that is retained in shell history: ```bash export PRODUCTION_DATABASE_MODE=external read -rsp 'External DATABASE_URL: ' PRODUCTION_DATABASE_URL && echo export PRODUCTION_DATABASE_URL ./scripts/init-production-env.sh whoneedhelp.com unset PRODUCTION_DATABASE_URL ``` Set `PRODUCTION_APP_TOPOLOGY=split` during generation when independent web and worker scaling is required. In external mode `scripts/compose-up.sh` builds the migration image, runs `scripts/check-database.sh`, and fails before migration or application startup unless PostgreSQL and `PostGIS_Version()` are reachable. The check also reports the observed TLS flag. The connection URL must follow the database provider's verified TLS/CA requirements. `scripts/compose.sh .env config --services` is the canonical read-only render check. In external mode its output must not contain `db`. Use `scripts/compose.sh .env ps -a` and `scripts/compose.sh .env logs web worker` for the selected mode. The repository's Compose backup, restore, rotation, and rollback-based staging scripts deliberately refuse `DATABASE_MODE=external`; they are scoped to the project-owned database container. For an external database, use the provider's verified backup/restore procedure and test recovery before public launch. ## Compose database backup Create a PostgreSQL 18 custom-format archive, validate its table of contents, and write a SHA-256 manifest: ```bash ./scripts/backup-compose.sh ``` The default destination is the ignored `output/backups/` directory. An explicit new destination may be supplied as the only argument. The command refuses to overwrite either an archive or its checksum manifest and writes through temporary files before publishing the final pair. Files and newly created directories are restricted by `umask 077`. The archive covers the configured application database. PostgreSQL cluster globals such as roles and tablespaces are not part of `pg_dump`; deployment credentials and database roles must be provisioned separately from secrets. Local backup files on the same workstation are not an off-site backup. ## Encrypted local S3-compatible backup drill The isolated load project can run a complete encrypted Restic/MinIO drill: ```bash ./scripts/load-stack-up.sh ./scripts/backup-s3-drill.sh local-encrypted-backup ``` The script refuses the staging Compose project and validates the project and service labels of every pre-existing container in its scope. On first use, `scripts/ensure-local-load-env.sh` generates independent random MinIO and Restic credentials in ignored `output/runtime/load.env` and restricts that file to mode `0600`. MinIO publishes Docker-assigned ports only on `127.0.0.1`; the observed API and console URLs are printed after a successful run. The backup tool combines the matching PostgreSQL 18 client with pinned Restic rebuilt on Go 1.26.5. MinIO server and client are also rebuilt as non-root Alpine images from checksum-pinned upstream source commits with the exact dependency updates recorded in `Dockerfile.minio`. The quality gate verifies their reported release, commit, Go runtime, configured user, and current HIGH/CRITICAL vulnerability scan. `restic backup --stdin-from-command` runs a custom-format `pg_dump`, checks the producer exit status, encrypts the data, and uploads it directly to MinIO. No plaintext database dump is written to the host. The drill then: 1. runs `restic check --read-data`; 2. streams `restic dump` into `pg_restore --list`; 3. restores into a uniquely named database created from `template0`; 4. checks tables, current Ecto migrations, PostGIS, and release migration readiness before removing that exact database; 5. clones and corrupts an isolated repository and requires both check and dump to fail; 6. stops an exact scoped in-progress backup container only after encrypted objects reach MinIO, requires zero published snapshots, prunes unreferenced packs, and rechecks the repository; 7. removes and verifies removal of the corruption/interruption buckets and requires source table counts to remain unchanged. The successful encrypted bucket is deliberately retained in the named local MinIO volume. Non-secret evidence is written under ignored `output/backups-s3//`; the runtime scratch directory is under ignored `tmp/backup-s3/`. Use a unique lowercase run label of at most 32 characters. The command refuses to replace an existing retained bucket. MinIO server and client are AGPLv3. `Dockerfile.minio` identifies the exact upstream source commits and contains the dependency changes and complete build commands used here. Before distributing or publicly operating modified images, review the license and make the corresponding source available as required; this runbook does not provide legal advice. This verifies encryption, local S3 protocol use, restore mechanics, and two failure paths on the observed workstation. A MinIO volume on that same workstation is not an off-site backup and does not establish production RPO, RTO, retention, capacity, key custody, object locking, or database HA. ## Local external-service boundary drill Run the OAuth, SMTP, and provider-neutral push protocol checks without public credentials, a server, or host-published ports: ```bash ./scripts/external-boundaries-run.sh local-boundaries ``` The script creates a uniquely named Compose project on an internal-only Docker network. It generates independent one-run OAuth and push credentials in an ignored mode-`0600` environment file, including a separate bearer token for the protected metrics endpoint, builds the production release plus a non-root standard-library Python protocol mock, and then verifies: `compose.external-boundaries.yaml` is the only deployment file that sets `ALLOW_INSECURE_EXTERNAL_HTTP=true`. Ordinary runtime configuration requires HTTPS for OAuth and push endpoints so provider credentials are not sent over plain HTTP. 1. GitHub-compatible OAuth authorization, PKCE S256, token exchange, normalized user lookup, state mismatch, provider rejection, one-time-code replay, a fresh flow after a temporary token error, and a token timeout; 2. SMTP acceptance, permanent recipient rejection without retry, one retry after a temporary greeting failure, a greeting timeout, and the result of submitting the same message twice; 3. the disabled default push boundary plus HTTP success, permanent rejection, one temporary retry, replay deduplication, and deduplication after an ambiguous timeout using the same idempotency key; 4. a fresh PostGIS database, current migrations, and two Oban worker replicas; request acceptance and new-chat domain transactions enqueue stable user-recipient events, replay is deduplicated before HTTP, an injected temporary chat delivery fails its first Oban attempt and completes on its second, and private message text is absent from the push payload. The evidence JSON and mock state contain counters, booleans, normalized identity fields, payload digests, run-scoped user/event identifiers, and the privacy-safe notification metadata asserted by the drill. The script fails if any generated secret appears in retained evidence. Its trap validates exact Compose labels, removes only that project, volumes and one-run images, and deletes the temporary credential file. Failure logs are retained under the same ignored evidence directory. This drill uses the application's real Assent/Req and Swoosh/gen_smtp clients, but the providers are local. It therefore verifies client-side protocol and product integration, not GitHub, SMTP-provider, FCM, or APNs availability. SMTP permits duplicate delivery after ambiguous outcomes, so the result explicitly makes no exactly-once claim. Push currently targets a stable `user:` recipient; selecting a provider, registering device tokens, and resolving that user to devices remain deployment/provider work. The implementation follows the configured adapter interfaces and protocol semantics documented by [Assent 0.3.1](https://hexdocs.pm/assent/0.3.1/Assent.HTTPAdapter.html), [GitHub OAuth](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps), and [SMTP RFC 5321](https://datatracker.ietf.org/doc/html/rfc5321). The application uses Swoosh's SMTP adapter and `gen_smtp`. Verify the selected relay's authenticated delivery, sender-domain authentication, and delivered message headers before launch. Test and production may share an account-level SMTP login only when they use independently revocable SMTP passwords. ## Isolated restore drill Run a real restore into a uniquely named temporary database: ```bash ./scripts/restore-drill-compose.sh output/backups/compose-YYYYMMDD-HHMMSS.dump ``` The drill: 1. validates the SHA-256 manifest; 2. validates the archive table of contents; 3. creates a pristine database from `template0`; 4. restores with `pg_restore --exit-on-error`; 5. reads every restored public application table and checks the PostGIS library; 6. runs the current immutable release's migrations and migration-readiness check against only the temporary database; 7. drops only the temporary drill database and verifies that it is gone. A trap also attempts to drop the exact temporary database if a check fails. The source application database is never passed to `pg_restore`, `dropdb`, a clean operation, or the drill migration runner. The drill intentionally does not compare an older backup's row counts to the live source, because concurrent legitimate writes or a historical archive would make that comparison invalid. ## Isolated Compose upgrade rehearsal After creating a current custom-format backup, run the complete current release against an isolated restored copy: ```bash ./scripts/upgrade-rehearsal-compose.sh \ output/backups/compose-YYYYMMDD-HHMMSS.dump ``` The rehearsal validates the checksum and archive catalog, reads the public origin configuration from ignored `.env`, and uses only the independently generated credentials in ignored mode-`0600` `output/runtime/e2e.env`. It builds a uniquely tagged production release, creates a uniquely named Compose project and database from `template0`, restores the archive, records application-table counts, and then: 1. applies every current timestamped Ecto migration; 2. requires the localized category column and all 11 valid cursor indexes; 3. starts 2 web and 2 worker replicas behind the isolated Traefik instance; 4. requires the four-node BEAM cluster and cross-node PubSub probe; 5. checks the production HTTP-to-HTTPS redirect, trusted-proxy public pages, and both health endpoints; 6. requires an empty before/after diff for every public application table except the expected migration and Oban-internal tables; 7. removes and verifies removal of the exact containers, networks, database volume, and one-run image. The input archive is read-only and is not copied into the evidence directory. The ordinary Compose project, source database, public route, and running containers are outside the generated project scope. Non-secret evidence is retained under ignored mode-`0700` `output/upgrade-rehearsal//`, with files mode `0600`. ## Clean tracked-revision deployment drill Run: ```bash ./scripts/clean-deploy-verify.sh ``` The script archives the current tracked Git revision into a new temporary directory. It verifies that the archive contains no local `.env`, Git metadata, generated output, or existing E2E/load environment. It generates independent one-run PostgreSQL, Phoenix, handover, BEAM-cookie, and metrics secrets with mode `0600`; optional GitHub OAuth and push delivery stay disabled. The drill uses a unique Compose project, application image, Docker socket proxy image, PostGIS image, Traefik image, router/service name, internal network, PostgreSQL volume, database, and dynamic host ports. It requires exactly two healthy web and two running worker replicas, all tracked migrations, seeded categories, an idempotent repeated migration, working readiness/home/registration/Mailpit routes, four connected BEAM nodes, and a cross-replica PubSub message. It then removes the exact project including volumes, all four one-run images, and the temporary archive, and verifies that every scoped resource is absent. If Compose validation fails before project creation, cleanup avoids invoking an unrenderable Compose configuration and still verifies that the generated scope is absent. Evidence is retained under `output/portability//`. This is a local Docker portability observation. It does not establish production SMTP, OAuth, push-provider, TLS, database-HA, storage, capacity, or jurisdictional readiness. ## Service checks ```bash docker compose ps curl --fail http://localhost:4010/healthz/live curl --fail http://localhost:4010/healthz/ready ./scripts/verify-realtime-cluster.sh compose ``` `live` verifies that the web process can serve HTTP. `ready` additionally runs `SELECT 1` through the configured Ecto repository. The cluster probe subscribes on one connected BEAM node and broadcasts from another. Health checks do not replace alerting, database backups, restore drills, or application-level synthetic checks. ## Local failure and rolling-replacement drills The isolated load project can exercise process crashes, sequential container replacement, and a real Oban retry without touching the normal Compose project: ```bash ./scripts/load-cycle.sh local-resilience resilience ``` The lifecycle wrapper assigns unique Compose and image names and removes that exact run's containers, networks, named volumes, temporary environment, and image tags on success, failure, or interruption. Use `load-stack-up.sh` plus `load-resilience-run.sh` only when the isolated stack must remain available for manual inspection. The resilience script refuses `LOAD_PROJECT=who_need_help` and verifies the Compose project/service labels of every container before stopping it. It: 1. continuously calls readiness through the isolated Traefik route; 2. terminates the BEAM process in one web and one worker container and requires Docker's observed restart count to increase; 3. removes and replaces each web and worker replica one at a time; 4. waits for every configured BEAM node, then runs the cross-node PubSub probe; 5. enqueues a side-effect-free local worker that fails its first Oban attempt and succeeds on its second; 6. removes that exact Oban row and requires no fixture domain rows to remain. Traefik's retry middleware is attached to the HTTP and local TLS routers. Its attempt count is an environment input. Traefik retries transport failures and, with the checked configuration, does not opt in to retrying non-idempotent requests. This reduces a stale-backend window; it is not a claim of production availability. For the project-owned kind cluster, run: ```bash ./scripts/kind-rolling-verify.sh local-kind-rollout ``` The kind control plane is an optional verification environment, not part of the ordinary public Compose route. Pause it without deleting its container or Kubernetes state when the cluster is not being tested: ```bash ./scripts/kind-stop.sh ``` The next `./scripts/kind-up.sh` validates the ownership marker and kind labels, starts the stopped control plane, then continues with image loading and chart reconciliation. That script requires both the kind ownership marker and the control-plane cluster label before invoking `rollout restart`. It changes only the web and worker Deployment pod templates. It snapshots application-table counts before and after, continuously probes the observed Docker mapping for the chart's NodePort, requires all four old pod UIDs to disappear, waits for the exact BEAM peer count, and verifies cross-node PubSub. PostGIS, its hostPath, the Kubernetes Secret, and the namespace are not recreated. The rollout timeout, probe interval/timeout/retry count, and cluster-join timeout are experiment inputs. They are not production SLOs or resource requirements. ## Isolated Oban burst measurement The worker role consumes only `maintenance` and `push`. Configure their per-worker limits with `OBAN_MAINTENANCE_CONCURRENCY` and `OBAN_PUSH_CONCURRENCY`; multiplying either value by the number of worker replicas gives the configured cluster-wide concurrency for that queue. After starting the isolated load project, run an explicitly sized experiment: ```bash ./scripts/load-stack-up.sh ./scripts/oban-burst-run.sh local-oban-burst 1000 120 ``` The three required arguments are an evidence label, job count, and recorded experiment timeout in seconds. They are not capacity thresholds. The script refuses the ordinary `who_need_help` project, verifies the expected worker replicas, inserts only confirmed `LocalBurstProbe` jobs with a unique run id, records each worker's effective queue configuration and container samples, requires every job to complete, compares domain-table row counts, and deletes exactly its own jobs. The probe worker has no product side effects and no product flow enqueues it. ## Kubernetes ingress isolation `networkPolicy.enabled=true` renders the chart's ingress NetworkPolicy. Enforcement is a property of the cluster CNI, not of the YAML object alone. Before relying on it, verify that the target cluster uses a NetworkPolicy- capable plugin and run positive HTTP/cluster checks plus negative blocked-port checks there. Egress remains intentionally unrestricted until the actual database and external-service destinations are known. ## BEAM runtime memory guard Compose sets `ERL_ZFLAGS="+Q ${ERLANG_PORT_LIMIT}"`; Helm renders the same flag from `app.erlangPortLimit`. Both default to `65536`, OTP's normal port-table limit. This makes the runtime independent of an unusually large host or nested container `nofile` limit. Validate all live replicas after deployment: ```bash ./scripts/verify-beam-runtime.sh compose ./scripts/verify-beam-runtime.sh kind ``` The command records the effective port count/limit, allocated port-table bytes, BEAM memory, process count, cgroup memory, and RSS for each web and worker replica. It fails when a live VM does not use the configured limit; kind mode also requires the desired number of Ready application pods. `65536` is a concurrency ceiling for simultaneously existing Erlang ports (files, sockets, and drivers), not a container memory limit. Do not lower or raise it from a RAM estimate alone. A changed value must be validated against measured peak port usage and the target environment. ## Protected Prometheus metrics The web role exposes Prometheus text format at `/metrics`. It requires the independent `METRICS_TOKEN` deployment secret: ```bash curl --fail \ --header "Authorization: Bearer $METRICS_TOKEN" \ http://localhost:4010/metrics ``` The endpoint returns `401` without the exact token, disables response caching, and does not put the credential in a URL. The reporter exports cumulative HTTP request and duration, router exception, database query and duration, WebSocket connection, VM memory, and scheduler run-queue metrics. Cumulative durations are integer microseconds because the selected reporter's sum accumulator is integer-based; divide by `1_000_000` in PromQL when seconds are required. Definitions intentionally have no request path, user, request, or event-name labels that could create unbounded cardinality. The locked `telemetry_metrics_prometheus_core` reporter aggregates distribution samples only when a scrape occurs. Each application VM therefore also runs a supervised internal aggregation every ten seconds. This keeps the reporter's raw distribution table independent of whether an external Prometheus server is currently configured, while preserving the cumulative histograms exposed by the authenticated endpoint. The generated exposition text from the internal aggregation is discarded. After a deployment, the raw table can be inspected without exposing metric credentials: ```bash /app/bin/who_need_help eval \ 'IO.inspect(:ets.info(:prometheus_metrics_dist, :size), label: "pending_distribution_samples")' ``` Under idle conditions the value returns to zero after the next aggregation interval. During traffic it represents only samples received since the most recent internal or external scrape; it is not the cumulative histogram count. Metrics are local to each BEAM process. Discover and scrape every web pod or container as a distinct target and preserve Prometheus's `instance` label. A request through the load-balanced public route reaches only one replica and is therefore useful as an authorization/smoke check, not as a cluster-wide aggregate. The isolated load project includes a local observability profile: ```bash ./scripts/load-stack-up.sh ./scripts/observability-run.sh local-observability ``` The run script refuses the staging project, validates every current web container's Compose labels, and writes a `file_sd` target for each observed internal IP. Prometheus reads the Bearer value from a mode-`0600` runtime file, not a tracked config or URL. Its direct request includes the internal `X-Forwarded-Proto: https` signal required by the application's production SSL rewrite while preserving the target's own `instance` label. Prometheus, Alertmanager, and Grafana are pinned by tag and digest. Their host ports default to Docker-assigned values bound only to `127.0.0.1`; the run prints the observed URLs. Grafana uses the random admin password generated in ignored `output/runtime/load.env`, disables anonymous signup, update checks, suggested plugin installation, and its unused built-in alert engine. The Prometheus datasource and ten-panel dashboard are provisioned from tracked files. The dashboard separates all discovered web and worker replicas and includes HTTP traffic/latency, BEAM memory and scheduler run queues, Ecto execution and pool-wait latency, and Oban outcomes and queue wait by queue. The verification stops exactly one scoped load web container. The `WhoNeedHelpWebReplicaUnavailable` rule is based only on the factual `up == 0` result; it is a local failure drill, not an invented latency, capacity, or production SLO threshold. The script requires both firing and resolved webhook payloads from Alertmanager, starts the same container, waits for every direct target, and compares read-only database counts before and after. Evidence is retained in `output/observability/` without the metrics or Grafana secrets. Stop only the monitoring services with: ```bash ./scripts/observability-stop.sh ``` Prometheus/Grafana/Alertmanager retention, production notification destinations, production availability, and measured alert policies remain deployment decisions. In Kubernetes, put the metrics token in `existingSecret`; configure the external scraper to send it as a Bearer token. ## Production release without pushing the frozen repository The post-submission workflow keeps the public Git repository and `test.whoneedhelp.com` untouched. A clean local commit is packaged as a verified Git bundle and transferred directly over SSH to only `/srv/who_need_help-production`: ```bash ./scripts/production-release.sh plan whoneedhelp ``` The default `plan` action is read-only. It verifies the exact local and remote commits, requires a fast-forward history, checks the production checkout, Compose scope and healthy containers, checks public readiness, opens a read-only PostgreSQL connection, and runs the server-release environment capability preflight. Before the first Google Play release, this preflight allows only the absent Play App Signing certificate; the stricter `check-environment-readiness.sh .env --require-release` remains the gate for publishing Android through Google Play. The plan neither uploads a bundle nor creates a backup. After reviewing the exact commit printed by the plan, execution additionally requires an explicit per-commit confirmation: ```bash WNH_PRODUCTION_RELEASE_CONFIRM=whoneedhelp.com:FULL_COMMIT \ ./scripts/production-release.sh apply whoneedhelp ``` The apply path refuses tracked local or remote modifications. It then: 1. creates and verifies a full Git bundle for exactly that clean commit; 2. uploads only that bundle to the production checkout's ignored `output/releases/`; 3. creates a custom-format PostgreSQL 18 backup without exposing the database password in process arguments; 4. verifies its archive catalog and SHA-256, then copies and verifies the backup again under local ignored `output/production-backups/`; 5. fast-forwards the production checkout without accessing or changing the test checkout or public remote; 6. selects immutable per-commit application image tags, applies migrations, starts only the application topology, and verifies public readiness through the already-running shared edge plus the Android App Links endpoints. Every newly added migration must have one reviewed entry in `priv/repo/migration_application_compatibility.tsv`. `application_safe` means the previously deployed application can run against the resulting schema; `forward_only` means that it cannot be guaranteed. The plan prints the aggregate policy. A forward-only apply requires a second exact confirmation: ```bash WNH_PRODUCTION_FORWARD_ONLY_CONFIRM=whoneedhelp.com:FULL_COMMIT:forward-only \ WNH_PRODUCTION_RELEASE_CONFIRM=whoneedhelp.com:FULL_COMMIT \ ./scripts/production-release.sh apply whoneedhelp ``` For a forward-only release, all candidate images are built first, the old application is stopped before migration begins, and the target application is started only after the migration runner succeeds. If anything fails after the migration begins, the release deliberately leaves the old application stopped and records that boundary in the release manifest. Restarting an older image against a potentially incompatible schema is never automatic. For an `application_safe` release, an application startup failure restores the previous immutable application image tags and attempts to recover public readiness through the unchanged edge. A `forward_only` release never starts the old application after migration begins. Neither path reverses Git source or Ecto migrations automatically. The per-release rollback manifest and backup paths are recorded below the production checkout's ignored `output/releases/`. The copied backup is separate from the production host, but a long-term encrypted off-site backup destination remains an operational requirement. ## Rollback boundary The release image is immutable and migrations run as a separate one-shot role. Before a schema rollout, create and restore-test a current backup. Application rollback and database migration rollback are separate decisions: do not run an Ecto down migration merely because an image is rolled back. Inspect the exact migration and compatibility boundary first. Every successful SSH release prints a mode-`0600` `rollback-manifest.txt`. Before changing production, inspect that exact manifest with the read-only rollback plan: ```bash ./scripts/production-rollback.sh plan \ /srv/who_need_help-production/output/releases/RELEASE_ID/rollback-manifest.txt \ whoneedhelp ``` The rollback plan refuses manifests marked `forward_only`. Such releases must be repaired forward after inspecting the exact migration state; an application image rollback is available only when the manifest records `migration_policy=application_safe`. The plan requires the manifest target to be the currently checked-out production commit, verifies the previous immutable application image still exists, checks the pre-release backup catalog and checksum, and prints the exact confirmation token. It does not change the remote environment or containers. After separately reviewing application/schema backward compatibility and approving that exact scope, run: ```bash WNH_PRODUCTION_ROLLBACK_CONFIRM=whoneedhelp.com:TARGET_COMMIT:PREVIOUS_COMMIT \ ./scripts/production-rollback.sh apply \ /srv/who_need_help-production/output/releases/RELEASE_ID/rollback-manifest.txt \ whoneedhelp ``` This application rollback atomically restores the three previous application image selectors and recreates only the selected application topology with `--no-build`. It never changes or recreates the shared edge. It verifies the resulting image identities, container health, public readiness through the unchanged edge, and App Links. A failed rollback attempts to restore the pre-rollback image selection. The Git checkout intentionally remains at the newer source commit so the reviewed release tooling and manifest remain available. The command never restores PostgreSQL, reverses Ecto migrations, changes the test deployment, or touches the public Git/Devpost submission. The repository intentionally does not ship an automatic destructive production database restore command.